2024-06-03T05:47:35.264 INFO:root:teuthology version: 0.0.1.dev303+g861a8dc 2024-06-03T05:47:35.264 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} --name teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi --verbose --archive /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738514 --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738514/orig.config.yaml 2024-06-03T05:47:35.285 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-03T05:47:35.358 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738514 branch: squid description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} email: ceph-qa@ceph.com first_in_suite: false job_id: '7738514' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5132 sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba sleep_before_teardown: 0 subset: 20949/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 targets: smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNUVbJUwc2gtyYrmmMsivjOp67ZjzFbAmc3JHZBCSOHv+avMMXccqlHVnD3wobAdd1yMKZpWo0BIDEh7W25YjGg= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 861a8dcf7aa816a26e13f039336f7ed0a9aec0fa timestamp: 2024-06-02_21:00:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-03T05:47:35.358 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa; will attempt to use it 2024-06-03T05:47:35.359 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks 2024-06-03T05:47:35.359 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-03T05:47:35.360 INFO:teuthology.task.internal:Checking packages... 2024-06-03T05:47:35.380 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d89accfe3201286f000e065a1faea2a88c7a6ba' 2024-06-03T05:47:35.380 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-03T05:47:35.380 INFO:teuthology.packaging:ref: None 2024-06-03T05:47:35.380 INFO:teuthology.packaging:tag: None 2024-06-03T05:47:35.380 INFO:teuthology.packaging:branch: squid 2024-06-03T05:47:35.381 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-03T05:47:35.381 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-03T05:47:35.597 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2952.gdfcc4b3c 2024-06-03T05:47:35.599 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-03T05:47:35.605 INFO:teuthology.task.internal:no buildpackages task found 2024-06-03T05:47:35.605 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-03T05:47:35.695 INFO:teuthology.task.internal:Saving configuration 2024-06-03T05:47:35.703 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-03T05:47:35.763 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-03T05:47:35.789 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738514', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-03 05:41:38.693056', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNUVbJUwc2gtyYrmmMsivjOp67ZjzFbAmc3JHZBCSOHv+avMMXccqlHVnD3wobAdd1yMKZpWo0BIDEh7W25YjGg='} 2024-06-03T05:47:35.789 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-03T05:47:35.826 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-06-03T05:47:35.826 INFO:teuthology.run_tasks:Running task console_log... 2024-06-03T05:47:35.929 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7f7e617ac0>, signals=[15]) 2024-06-03T05:47:35.929 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-03T05:47:35.935 INFO:teuthology.task.internal:Opening connections... 2024-06-03T05:47:35.935 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2024-06-03T05:47:35.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T05:47:36.011 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-03T05:47:36.018 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2024-06-03T05:47:36.100 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2024-06-03T05:47:36.100 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2024-06-03T05:47:36.157 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="CentOS Stream" 2024-06-03T05:47:36.158 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="9" 2024-06-03T05:47:36.158 INFO:teuthology.orchestra.run.smithi039.stdout:ID="centos" 2024-06-03T05:47:36.158 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="rhel fedora" 2024-06-03T05:47:36.158 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="9" 2024-06-03T05:47:36.158 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el9" 2024-06-03T05:47:36.158 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-03T05:47:36.158 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2024-06-03T05:47:36.158 INFO:teuthology.orchestra.run.smithi039.stdout:LOGO="fedora-logo-icon" 2024-06-03T05:47:36.159 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-03T05:47:36.159 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://centos.org/" 2024-06-03T05:47:36.159 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-03T05:47:36.159 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-03T05:47:36.159 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-03T05:47:36.159 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2024-06-03T05:47:36.180 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-03T05:47:36.187 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-03T05:47:36.209 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-03T05:47:36.209 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2024-06-03T05:47:36.226 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-03T05:47:36.272 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-03T05:47:36.272 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2024-06-03T05:47:36.295 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-03T05:47:36.375 INFO:teuthology.run_tasks:Running task kernel... 2024-06-03T05:47:36.407 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-03T05:47:36.407 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-03T05:47:36.407 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-03T05:47:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T05:47:36.425 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-06-03T05:47:36.483 INFO:teuthology.orchestra.run.smithi039.stdout:5.14.0-437.el9.x86_64 2024-06-03T05:47:36.483 INFO:teuthology.task.kernel:Running kernel on smithi039: 5.14.0-437.el9.x86_64 2024-06-03T05:47:36.483 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum install -y kernel 2024-06-03T05:47:38.501 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.1 MB 00:00 2024-06-03T05:47:40.244 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 32 MB/s | 19 MB 00:00 2024-06-03T05:47:44.411 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.2 MB 00:00 2024-06-03T05:47:45.820 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 88 kB/s | 17 kB 00:00 2024-06-03T05:47:46.630 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 32 MB/s | 21 MB 00:00 2024-06-03T05:47:52.021 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-06-03T05:47:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-03T05:47:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-03T05:47:54.212 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:47:54.216 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:47:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-06-03T05:47:54.217 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:47:54.217 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:47:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-03T05:47:54.217 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-06-03T05:47:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-03T05:47:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-03T05:47:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-03T05:47:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:47:54.217 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:47:54.217 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:47:54.217 INFO:teuthology.orchestra.run.smithi039.stdout:Install 4 Packages 2024-06-03T05:47:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:47:54.218 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 80 M 2024-06-03T05:47:54.218 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 124 M 2024-06-03T05:47:54.218 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:47:54.539 INFO:teuthology.orchestra.run.smithi039.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 3.1 MB/s | 893 kB 00:00 2024-06-03T05:47:55.431 INFO:teuthology.orchestra.run.smithi039.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 14 MB/s | 16 MB 00:01 2024-06-03T05:47:56.048 INFO:teuthology.orchestra.run.smithi039.stdout:(3/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 19 MB/s | 35 MB 00:01 2024-06-03T05:47:56.315 INFO:teuthology.orchestra.run.smithi039.stdout:(4/4): kernel-modules-core-5.14.0-452.el9.x86_6 16 MB/s | 29 MB 00:01 2024-06-03T05:47:56.315 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:47:56.316 INFO:teuthology.orchestra.run.smithi039.stdout:Total 38 MB/s | 80 MB 00:02 2024-06-03T05:47:57.041 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:47:57.465 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:47:57.465 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:47:58.475 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:47:58.476 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:48:01.003 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:48:01.675 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-03T05:48:01.790 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-03T05:48:04.559 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-03T05:48:04.708 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-03T05:48:09.814 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-03T05:48:09.872 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:48:14.984 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:48:45.967 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:48:46.010 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:48:46.438 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:48:46.438 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-03T05:48:46.438 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-03T05:48:46.438 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-03T05:48:47.302 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:48:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:48:47.303 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:48:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-03T05:48:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-03T05:48:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-03T05:48:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-03T05:48:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:48:47.303 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:48:47.656 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2024-06-03T05:48:47.741 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-03T05:48:47.742 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-03T05:48:47.743 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-03T05:48:47.743 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-03T05:48:47.743 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-03T05:48:47.743 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-06-03T05:48:47.743 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=None, version=distro) 2024-06-03T05:48:47.743 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2024-06-03T05:48:47.798 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-03T05:48:47.798 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-03T05:48:47.798 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -qi grub2-tools 2024-06-03T05:48:47.839 INFO:teuthology.orchestra.run.smithi039.stdout:Name : grub2-tools 2024-06-03T05:48:47.840 INFO:teuthology.orchestra.run.smithi039.stdout:Epoch : 1 2024-06-03T05:48:47.840 INFO:teuthology.orchestra.run.smithi039.stdout:Version : 2.06 2024-06-03T05:48:47.840 INFO:teuthology.orchestra.run.smithi039.stdout:Release : 68.el9 2024-06-03T05:48:47.840 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: x86_64 2024-06-03T05:48:47.840 INFO:teuthology.orchestra.run.smithi039.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-03T05:48:47.840 INFO:teuthology.orchestra.run.smithi039.stdout:Group : Unspecified 2024-06-03T05:48:47.840 INFO:teuthology.orchestra.run.smithi039.stdout:Size : 8264254 2024-06-03T05:48:47.840 INFO:teuthology.orchestra.run.smithi039.stdout:License : GPLv3+ 2024-06-03T05:48:47.840 INFO:teuthology.orchestra.run.smithi039.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-03T05:48:47.840 INFO:teuthology.orchestra.run.smithi039.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-03T05:48:47.840 INFO:teuthology.orchestra.run.smithi039.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-03T05:48:47.840 INFO:teuthology.orchestra.run.smithi039.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-03T05:48:47.841 INFO:teuthology.orchestra.run.smithi039.stdout:Packager : builder@centos.org 2024-06-03T05:48:47.841 INFO:teuthology.orchestra.run.smithi039.stdout:Vendor : CentOS 2024-06-03T05:48:47.841 INFO:teuthology.orchestra.run.smithi039.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-03T05:48:47.841 INFO:teuthology.orchestra.run.smithi039.stdout:Summary : Support tools for GRUB. 2024-06-03T05:48:47.841 INFO:teuthology.orchestra.run.smithi039.stdout:Description : 2024-06-03T05:48:47.841 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:48:47.841 INFO:teuthology.orchestra.run.smithi039.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-03T05:48:47.841 INFO:teuthology.orchestra.run.smithi039.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-03T05:48:47.841 INFO:teuthology.orchestra.run.smithi039.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-03T05:48:47.841 INFO:teuthology.orchestra.run.smithi039.stdout:hardware devices. 2024-06-03T05:48:47.842 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:48:47.842 INFO:teuthology.orchestra.run.smithi039.stdout:This subpackage provides tools for support of all platforms. 2024-06-03T05:48:47.842 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-03T05:48:47.842 INFO:teuthology.task.kernel:Updating grub on smithi039 to boot 5.14.0-452.el9.x86_64 2024-06-03T05:48:47.843 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-03T05:48:48.536 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2024-06-03T05:48:50.021 INFO:teuthology.orchestra.run.smithi039.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-03T05:48:50.061 INFO:teuthology.orchestra.run.smithi039.stderr:done 2024-06-03T05:48:50.064 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-06-03T05:48:50.093 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.7ZD1gmaV05 2024-06-03T05:48:50.093 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7ZD1gmaV05 2024-06-03T05:48:50.159 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.7ZD1gmaV05 2024-06-03T05:48:50.294 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.7ZD1gmaV05 is 6KB 2024-06-03T05:48:50.342 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.7ZD1gmaV05 2024-06-03T05:48:50.356 DEBUG:teuthology.orchestra.run.smithi039:> sudo /bin/ls /boot/loader/entries || true 2024-06-03T05:48:50.419 INFO:teuthology.orchestra.run.smithi039.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-03T05:48:50.420 INFO:teuthology.orchestra.run.smithi039.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-03T05:48:50.420 INFO:teuthology.orchestra.run.smithi039.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-03T05:48:50.420 INFO:teuthology.orchestra.run.smithi039.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-03T05:48:50.421 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-03T05:48:50.564 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2024-06-03T05:49:20.594 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-06-03T05:49:20.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-06-03T05:49:20.596 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T05:49:55.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-06-03T05:50:04.110 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-06-03T05:50:04.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T05:50:07.166 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-06-03T05:50:19.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-06-03T05:50:19.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T05:51:19.238 DEBUG:teuthology.orchestra.remote:timed out 2024-06-03T05:51:34.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-06-03T05:51:34.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T05:51:34.620 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-06-03T05:51:35.135 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-06-03T05:51:35.136 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-452.el9.x86_64"... 2024-06-03T05:51:35.136 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-06-03T05:51:35.176 INFO:teuthology.orchestra.run.smithi039.stdout:5.14.0-452.el9.x86_64 2024-06-03T05:51:35.176 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-03T05:51:35.176 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-03T05:51:35.177 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-03T05:51:36.178 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-06-03T05:51:36.178 DEBUG:teuthology.orchestra.run.smithi039:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-03T05:51:36.283 INFO:teuthology.orchestra.run.smithi039.stdout:ttyS1 2024-06-03T05:51:36.316 DEBUG:teuthology.parallel:result is None 2024-06-03T05:51:36.316 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-03T05:51:36.324 INFO:teuthology.task.internal:Creating test directory... 2024-06-03T05:51:36.324 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-03T05:51:36.340 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-03T05:51:36.345 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-03T05:51:36.353 INFO:teuthology.task.internal:Creating archive directory... 2024-06-03T05:51:36.353 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-03T05:51:36.404 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-03T05:51:36.411 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-03T05:51:36.411 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-03T05:51:36.468 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:51:36.478 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:51:36.480 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-03T05:51:36.489 INFO:teuthology.task.internal:Configuring sudo... 2024-06-03T05:51:36.489 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-03T05:51:36.547 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-03T05:51:36.555 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-03T05:51:36.556 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-03T05:51:36.602 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-03T05:51:36.686 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-03T05:51:36.849 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-03T05:51:36.891 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-03T05:51:36.932 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-06-03T05:51:36.932 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-03T05:51:36.998 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2024-06-03T05:51:37.074 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-03T05:51:37.385 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-03T05:51:37.394 INFO:teuthology.task.internal:Starting timer... 2024-06-03T05:51:37.394 INFO:teuthology.run_tasks:Running task pcp... 2024-06-03T05:51:37.404 INFO:teuthology.run_tasks:Running task selinux... 2024-06-03T05:51:37.412 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-06-03T05:51:37.413 DEBUG:teuthology.orchestra.run.smithi039:> sudo service auditd rotate 2024-06-03T05:51:37.489 INFO:teuthology.orchestra.run.smithi039.stdout:Rotating logs: 2024-06-03T05:51:37.490 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-03T05:51:37.490 DEBUG:teuthology.orchestra.run.smithi039:> /usr/sbin/getenforce 2024-06-03T05:51:37.510 INFO:teuthology.orchestra.run.smithi039.stdout:Permissive 2024-06-03T05:51:37.510 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi039.front.sepia.ceph.com': 'permissive'} 2024-06-03T05:51:37.510 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-03T05:51:37.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T05:51:37.576 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-03T05:51:37.576 DEBUG:teuthology.orchestra.run.smithi039:> sudo /usr/sbin/setenforce permissive 2024-06-03T05:51:37.642 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-03T05:51:37.654 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-03T05:51:37.665 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-03T05:51:37.666 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi039.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-03T05:56:02.496 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi039.front.sepia.ceph.com')] 2024-06-03T05:56:02.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-06-03T05:56:02.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T05:56:02.574 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-06-03T05:56:02.662 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-06-03T05:56:02.663 INFO:teuthology.run_tasks:Running task clock... 2024-06-03T05:56:02.675 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-03T05:56:02.676 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-03T05:56:02.676 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-03T05:56:02.733 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-03T05:56:02.748 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-03T05:56:02.773 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: ntpd: command not found 2024-06-03T05:56:02.785 INFO:teuthology.orchestra.run.smithi039.stdout:506 Cannot talk to daemon 2024-06-03T05:56:02.799 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-03T05:56:02.814 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-03T05:56:02.877 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2024-06-03T05:56:02.880 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-03T05:56:02.881 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2024-06-03T05:56:02.881 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T05:56:02.881 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T05:56:02.881 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T05:56:02.881 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T05:56:02.882 INFO:teuthology.run_tasks:Running task install... 2024-06-03T05:56:02.892 DEBUG:teuthology.task.install:project ceph 2024-06-03T05:56:02.892 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'}} 2024-06-03T05:56:02.893 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-03T05:56:02.893 INFO:teuthology.task.install:Using flavor: default 2024-06-03T05:56:02.903 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-03T05:56:02.903 INFO:teuthology.task.install:extra packages: [] 2024-06-03T05:56:02.904 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-03T05:56:02.905 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-03T05:56:03.056 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/ 2024-06-03T05:56:03.056 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2946.g3d89accf 2024-06-03T05:56:03.360 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-03T05:56:03.360 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-06-03T05:56:03.360 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-03T05:56:03.388 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-03T05:56:03.389 DEBUG:teuthology.orchestra.run.smithi039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d89accfe3201286f000e065a1faea2a88c7a6ba/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-03T05:56:03.456 DEBUG:teuthology.orchestra.run.smithi039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-03T05:56:03.533 DEBUG:teuthology.orchestra.run.smithi039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-03T05:56:03.598 INFO:teuthology.orchestra.run.smithi039.stdout:check_obsoletes = 1 2024-06-03T05:56:03.599 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum clean all 2024-06-03T05:56:03.843 INFO:teuthology.orchestra.run.smithi039.stdout:45 files removed 2024-06-03T05:56:03.871 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-radosgw 2024-06-03T05:56:04.646 INFO:teuthology.orchestra.run.smithi039.stdout:ceph packages for x86_64 172 kB/s | 83 kB 00:00 2024-06-03T05:56:05.055 INFO:teuthology.orchestra.run.smithi039.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-06-03T05:56:05.538 INFO:teuthology.orchestra.run.smithi039.stdout:ceph source packages 4.3 kB/s | 1.9 kB 00:00 2024-06-03T05:56:06.330 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.1 MB 00:00 2024-06-03T05:56:07.998 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 37 MB/s | 19 MB 00:00 2024-06-03T05:56:12.074 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.2 MB 00:00 2024-06-03T05:56:13.508 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 80 kB/s | 17 kB 00:00 2024-06-03T05:56:14.635 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-06-03T05:56:19.976 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-06-03T05:56:22.580 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:56:22.593 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:56:22.593 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-06-03T05:56:22.593 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:56:22.593 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:56:22.593 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 10 M 2024-06-03T05:56:22.593 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2024-06-03T05:56:22.593 INFO:teuthology.orchestra.run.smithi039.stdout: librados2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.4 M 2024-06-03T05:56:22.594 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-03T05:56:22.594 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-06-03T05:56:22.594 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-03T05:56:22.594 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.4 M 2024-06-03T05:56:22.594 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 21 M 2024-06-03T05:56:22.594 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 25 k 2024-06-03T05:56:22.594 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-03T05:56:22.595 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-03T05:56:22.595 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-03T05:56:22.595 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 747 k 2024-06-03T05:56:22.595 INFO:teuthology.orchestra.run.smithi039.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-03T05:56:22.595 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-03T05:56:22.595 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-03T05:56:22.595 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 501 k 2024-06-03T05:56:22.596 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-03T05:56:22.596 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.3 M 2024-06-03T05:56:22.596 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-03T05:56:22.596 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-03T05:56:22.596 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-03T05:56:22.596 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 45 k 2024-06-03T05:56:22.596 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 135 k 2024-06-03T05:56:22.597 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 162 k 2024-06-03T05:56:22.597 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-03T05:56:22.597 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 321 k 2024-06-03T05:56:22.597 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 300 k 2024-06-03T05:56:22.597 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 99 k 2024-06-03T05:56:22.597 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-03T05:56:22.598 INFO:teuthology.orchestra.run.smithi039.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-03T05:56:22.599 INFO:teuthology.orchestra.run.smithi039.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-03T05:56:22.599 INFO:teuthology.orchestra.run.smithi039.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-03T05:56:22.600 INFO:teuthology.orchestra.run.smithi039.stdout:Installing weak dependencies: 2024-06-03T05:56:22.600 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-03T05:56:22.600 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:56:22.600 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:56:22.600 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:56:22.600 INFO:teuthology.orchestra.run.smithi039.stdout:Install 30 Packages 2024-06-03T05:56:22.600 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 2 Packages 2024-06-03T05:56:22.600 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:56:22.601 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 60 M 2024-06-03T05:56:22.601 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:56:23.386 INFO:teuthology.orchestra.run.smithi039.stdout:(1/32): ceph-base-19.0.0-2946.g3d89accf.el9.x86 8.5 MB/s | 5.4 MB 00:00 2024-06-03T05:56:23.428 INFO:teuthology.orchestra.run.smithi039.stdout:(2/32): ceph-selinux-19.0.0-2946.g3d89accf.el9. 605 kB/s | 25 kB 00:00 2024-06-03T05:56:23.528 INFO:teuthology.orchestra.run.smithi039.stdout:(3/32): libcephfs2-19.0.0-2946.g3d89accf.el9.x8 7.3 MB/s | 747 kB 00:00 2024-06-03T05:56:23.620 INFO:teuthology.orchestra.run.smithi039.stdout:(4/32): libradosstriper1-19.0.0-2946.g3d89accf. 5.3 MB/s | 501 kB 00:00 2024-06-03T05:56:24.129 INFO:teuthology.orchestra.run.smithi039.stdout:(5/32): librgw2-19.0.0-2946.g3d89accf.el9.x86_6 10 MB/s | 5.3 MB 00:00 2024-06-03T05:56:24.187 INFO:teuthology.orchestra.run.smithi039.stdout:(6/32): python3-ceph-argparse-19.0.0-2946.g3d89 775 kB/s | 45 kB 00:00 2024-06-03T05:56:24.329 INFO:teuthology.orchestra.run.smithi039.stdout:(7/32): ceph-radosgw-19.0.0-2946.g3d89accf.el9. 6.5 MB/s | 10 MB 00:01 2024-06-03T05:56:24.354 INFO:teuthology.orchestra.run.smithi039.stdout:(8/32): python3-ceph-common-19.0.0-2946.g3d89ac 809 kB/s | 135 kB 00:00 2024-06-03T05:56:24.413 INFO:teuthology.orchestra.run.smithi039.stdout:(9/32): python3-rados-19.0.0-2946.g3d89accf.el9 5.4 MB/s | 321 kB 00:00 2024-06-03T05:56:24.438 INFO:teuthology.orchestra.run.smithi039.stdout:(10/32): python3-cephfs-19.0.0-2946.g3d89accf.e 1.5 MB/s | 162 kB 00:00 2024-06-03T05:56:24.472 INFO:teuthology.orchestra.run.smithi039.stdout:(11/32): python3-rbd-19.0.0-2946.g3d89accf.el9. 5.0 MB/s | 300 kB 00:00 2024-06-03T05:56:24.514 INFO:teuthology.orchestra.run.smithi039.stdout:(12/32): python3-rgw-19.0.0-2946.g3d89accf.el9. 1.3 MB/s | 99 kB 00:00 2024-06-03T05:56:24.614 INFO:teuthology.orchestra.run.smithi039.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 753 kB/s | 107 kB 00:00 2024-06-03T05:56:24.656 INFO:teuthology.orchestra.run.smithi039.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-06-03T05:56:24.690 INFO:teuthology.orchestra.run.smithi039.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 912 kB/s | 160 kB 00:00 2024-06-03T05:56:24.923 INFO:teuthology.orchestra.run.smithi039.stdout:(16/32): ceph-common-19.0.0-2946.g3d89accf.el9. 9.7 MB/s | 21 MB 00:02 2024-06-03T05:56:24.957 INFO:teuthology.orchestra.run.smithi039.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.2 MB/s | 662 kB 00:00 2024-06-03T05:56:24.982 INFO:teuthology.orchestra.run.smithi039.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-06-03T05:56:25.007 INFO:teuthology.orchestra.run.smithi039.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-06-03T05:56:25.033 INFO:teuthology.orchestra.run.smithi039.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 712 kB/s | 36 kB 00:00 2024-06-03T05:56:25.058 INFO:teuthology.orchestra.run.smithi039.stdout:(21/32): python3-prettytable-0.7.2-27.el9.noarc 315 kB/s | 42 kB 00:00 2024-06-03T05:56:25.083 INFO:teuthology.orchestra.run.smithi039.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-06-03T05:56:25.342 INFO:teuthology.orchestra.run.smithi039.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 100 kB/s | 26 kB 00:00 2024-06-03T05:56:25.542 INFO:teuthology.orchestra.run.smithi039.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 243 kB/s | 49 kB 00:00 2024-06-03T05:56:25.601 INFO:teuthology.orchestra.run.smithi039.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 544 kB/s | 309 kB 00:00 2024-06-03T05:56:25.718 INFO:teuthology.orchestra.run.smithi039.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 384 kB/s | 67 kB 00:00 2024-06-03T05:56:25.851 INFO:teuthology.orchestra.run.smithi039.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.3 MB/s | 839 kB 00:00 2024-06-03T05:56:25.918 INFO:teuthology.orchestra.run.smithi039.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 956 kB/s | 192 kB 00:00 2024-06-03T05:56:25.994 INFO:teuthology.orchestra.run.smithi039.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2024-06-03T05:56:26.085 INFO:teuthology.orchestra.run.smithi039.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-06-03T05:56:26.327 INFO:teuthology.orchestra.run.smithi039.stdout:(31/32): librados2-19.0.0-2946.g3d89accf.el9.x8 8.3 MB/s | 3.4 MB 00:00 2024-06-03T05:56:26.419 INFO:teuthology.orchestra.run.smithi039.stdout:(32/32): librbd1-19.0.0-2946.g3d89accf.el9.x86_ 7.4 MB/s | 3.1 MB 00:00 2024-06-03T05:56:26.421 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:56:26.422 INFO:teuthology.orchestra.run.smithi039.stdout:Total 16 MB/s | 60 MB 00:03 2024-06-03T05:56:26.529 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:56:26.571 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:56:26.572 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:56:27.159 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:56:27.159 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:56:28.728 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:56:28.811 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-06-03T05:56:28.850 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-06-03T05:56:28.884 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-06-03T05:56:28.906 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-03T05:56:28.931 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-03T05:56:29.075 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-03T05:56:29.147 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-03T05:56:29.497 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-06-03T05:56:29.609 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 7/34 2024-06-03T05:56:29.703 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 7/34 2024-06-03T05:56:29.796 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 8/34 2024-06-03T05:56:29.819 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/34 2024-06-03T05:56:29.891 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/34 2024-06-03T05:56:29.929 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-06-03T05:56:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9. 11/34 2024-06-03T05:56:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 12/34 2024-06-03T05:56:30.072 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_6 13/34 2024-06-03T05:56:30.145 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_6 13/34 2024-06-03T05:56:30.179 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-06-03T05:56:30.292 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-06-03T05:56:30.336 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-06-03T05:56:30.380 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-06-03T05:56:30.411 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-06-03T05:56:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-06-03T05:56:30.474 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-06-03T05:56:30.805 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-06-03T05:56:30.853 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 22/34 2024-06-03T05:56:30.936 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 22/34 2024-06-03T05:56:30.967 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 23/34 2024-06-03T05:56:31.012 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-06-03T05:56:31.518 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-06-03T05:56:31.547 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 26/34 2024-06-03T05:56:31.618 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 26/34 2024-06-03T05:56:31.699 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 27/34 2024-06-03T05:56:31.741 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x8 28/34 2024-06-03T05:56:34.097 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 29/34 2024-06-03T05:56:34.138 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 29/34 2024-06-03T05:56:34.727 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 29/34 2024-06-03T05:56:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 30/34 2024-06-03T05:56:34.867 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 30/34 2024-06-03T05:56:34.868 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-03T05:56:34.868 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-03T05:56:34.868 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:56:34.906 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 31/34 2024-06-03T05:56:45.445 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 31/34 2024-06-03T05:56:45.446 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /sys 2024-06-03T05:56:45.446 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /proc 2024-06-03T05:56:45.446 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /mnt 2024-06-03T05:56:45.446 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /var/tmp 2024-06-03T05:56:45.446 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /home 2024-06-03T05:56:45.446 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /root 2024-06-03T05:56:45.446 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /tmp 2024-06-03T05:56:45.446 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:56:46.438 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 32/34 2024-06-03T05:56:46.471 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 32/34 2024-06-03T05:56:46.471 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:56:46.472 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-03T05:56:46.472 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-03T05:56:46.472 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-03T05:56:46.472 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:56:46.473 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-03T05:56:46.557 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-03T05:56:46.557 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-03T05:56:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-03T05:56:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 1/34 2024-06-03T05:56:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2/34 2024-06-03T05:56:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 3/34 2024-06-03T05:56:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 4/34 2024-06-03T05:56:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 5/34 2024-06-03T05:56:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_6 6/34 2024-06-03T05:56:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 7/34 2024-06-03T05:56:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9. 8/34 2024-06-03T05:56:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x8 9/34 2024-06-03T05:56:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 10/34 2024-06-03T05:56:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 11/34 2024-06-03T05:56:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 12/34 2024-06-03T05:56:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 13/34 2024-06-03T05:56:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-06-03T05:56:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-06-03T05:56:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-06-03T05:56:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-06-03T05:56:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-06-03T05:56:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-06-03T05:56:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-06-03T05:56:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-06-03T05:56:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-06-03T05:56:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-06-03T05:56:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-06-03T05:56:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-06-03T05:56:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-06-03T05:56:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-06-03T05:56:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-06-03T05:56:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-06-03T05:56:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-06-03T05:56:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 31/34 2024-06-03T05:56:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-06-03T05:56:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 33/34 2024-06-03T05:56:48.641 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-06-03T05:56:48.641 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:56:48.641 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2024-06-03T05:56:48.641 INFO:teuthology.orchestra.run.smithi039.stdout: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:56:48.641 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:56:48.641 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:56:48.641 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-03T05:56:48.641 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:56:48.641 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:56:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:56:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:56:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-03T05:56:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-03T05:56:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-03T05:56:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:56:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-03T05:56:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-03T05:56:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-03T05:56:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:56:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-03T05:56:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:56:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-03T05:56:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-03T05:56:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-03T05:56:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:56:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:56:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:56:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-03T05:56:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:56:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:56:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:56:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-03T05:56:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-03T05:56:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-03T05:56:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: re2-1:20211101-3.el9.x86_64 2024-06-03T05:56:48.644 INFO:teuthology.orchestra.run.smithi039.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-03T05:56:48.644 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:56:48.644 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:56:49.017 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-test 2024-06-03T05:56:49.612 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:30 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:56:49.737 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:56:49.737 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:56:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-06-03T05:56:49.738 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:56:49.738 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:56:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 48 M 2024-06-03T05:56:49.738 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-06-03T05:56:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 163 k 2024-06-03T05:56:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-03T05:56:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-03T05:56:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:56:49.738 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:56:49.738 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:56:49.738 INFO:teuthology.orchestra.run.smithi039.stdout:Install 4 Packages 2024-06-03T05:56:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:56:49.739 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 49 M 2024-06-03T05:56:49.739 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 207 M 2024-06-03T05:56:49.739 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:56:50.019 INFO:teuthology.orchestra.run.smithi039.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 305 kB 00:00 2024-06-03T05:56:50.061 INFO:teuthology.orchestra.run.smithi039.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-06-03T05:56:50.103 INFO:teuthology.orchestra.run.smithi039.stdout:(3/4): libcephsqlite-19.0.0-2946.g3d89accf.el9. 660 kB/s | 163 kB 00:00 2024-06-03T05:56:52.612 INFO:teuthology.orchestra.run.smithi039.stdout:(4/4): ceph-test-19.0.0-2946.g3d89accf.el9.x86_ 18 MB/s | 48 MB 00:02 2024-06-03T05:56:52.614 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:56:52.614 INFO:teuthology.orchestra.run.smithi039.stdout:Total 17 MB/s | 49 MB 00:02 2024-06-03T05:56:52.642 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:56:52.665 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:56:52.665 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:56:53.052 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:56:53.052 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:56:53.715 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:56:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-03T05:56:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-06-03T05:56:53.872 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 3/4 2024-06-03T05:56:58.698 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 3/4 2024-06-03T05:56:58.781 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 4/4 2024-06-03T05:57:01.164 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 4/4 2024-06-03T05:57:01.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 1/4 2024-06-03T05:57:01.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 2/4 2024-06-03T05:57:01.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-06-03T05:57:01.477 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-03T05:57:01.477 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:01.477 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:57:01.477 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:57:01.477 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:57:01.477 INFO:teuthology.orchestra.run.smithi039.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-03T05:57:01.477 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-03T05:57:01.477 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:01.477 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:57:01.774 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph 2024-06-03T05:57:02.373 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:43 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:57:02.503 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:57:02.506 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:02.507 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-06-03T05:57:02.507 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:02.507 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:57:02.507 INFO:teuthology.orchestra.run.smithi039.stdout: ceph x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 6.6 k 2024-06-03T05:57:02.507 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-06-03T05:57:02.507 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 2.4 M 2024-06-03T05:57:02.507 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.5 M 2024-06-03T05:57:02.507 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 249 k 2024-06-03T05:57:02.507 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 4.7 M 2024-06-03T05:57:02.507 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 17 M 2024-06-03T05:57:02.507 INFO:teuthology.orchestra.run.smithi039.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-03T05:57:02.507 INFO:teuthology.orchestra.run.smithi039.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-03T05:57:02.507 INFO:teuthology.orchestra.run.smithi039.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-03T05:57:02.507 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-03T05:57:02.508 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-03T05:57:02.508 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-03T05:57:02.508 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-03T05:57:02.508 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-03T05:57:02.508 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-03T05:57:02.508 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-03T05:57:02.508 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections 2024-06-03T05:57:02.508 INFO:teuthology.orchestra.run.smithi039.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-03T05:57:02.508 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-03T05:57:02.508 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-03T05:57:02.508 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-03T05:57:02.508 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-03T05:57:02.508 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-03T05:57:02.508 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:57:02.509 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:02.510 INFO:teuthology.orchestra.run.smithi039.stdout:Install 36 Packages 2024-06-03T05:57:02.510 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:02.510 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 30 M 2024-06-03T05:57:02.510 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 107 M 2024-06-03T05:57:02.510 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:57:03.155 INFO:teuthology.orchestra.run.smithi039.stdout:(1/36): ceph-19.0.0-2946.g3d89accf.el9.x86_64.r 37 kB/s | 6.6 kB 00:00 2024-06-03T05:57:03.513 INFO:teuthology.orchestra.run.smithi039.stdout:(2/36): ceph-mds-19.0.0-2946.g3d89accf.el9.x86_ 4.4 MB/s | 2.4 MB 00:00 2024-06-03T05:57:03.630 INFO:teuthology.orchestra.run.smithi039.stdout:(3/36): ceph-mgr-19.0.0-2946.g3d89accf.el9.x86_ 2.3 MB/s | 1.5 MB 00:00 2024-06-03T05:57:03.747 INFO:teuthology.orchestra.run.smithi039.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2946.g3d89 2.1 MB/s | 249 kB 00:00 2024-06-03T05:57:03.806 INFO:teuthology.orchestra.run.smithi039.stdout:(5/36): ceph-mon-19.0.0-2946.g3d89accf.el9.x86_ 7.2 MB/s | 4.7 MB 00:00 2024-06-03T05:57:03.906 INFO:teuthology.orchestra.run.smithi039.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-06-03T05:57:03.948 INFO:teuthology.orchestra.run.smithi039.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-06-03T05:57:03.990 INFO:teuthology.orchestra.run.smithi039.stdout:(8/36): python3-cryptography-36.0.1-4.el9.x86_6 6.7 MB/s | 1.2 MB 00:00 2024-06-03T05:57:04.015 INFO:teuthology.orchestra.run.smithi039.stdout:(9/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-06-03T05:57:04.040 INFO:teuthology.orchestra.run.smithi039.stdout:(10/36): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-06-03T05:57:04.066 INFO:teuthology.orchestra.run.smithi039.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-06-03T05:57:04.166 INFO:teuthology.orchestra.run.smithi039.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-06-03T05:57:04.208 INFO:teuthology.orchestra.run.smithi039.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-06-03T05:57:04.225 INFO:teuthology.orchestra.run.smithi039.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 592 kB/s | 35 kB 00:00 2024-06-03T05:57:04.250 INFO:teuthology.orchestra.run.smithi039.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-06-03T05:57:04.275 INFO:teuthology.orchestra.run.smithi039.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 831 kB/s | 42 kB 00:00 2024-06-03T05:57:04.303 INFO:teuthology.orchestra.run.smithi039.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 425 kB/s | 22 kB 00:00 2024-06-03T05:57:04.453 INFO:teuthology.orchestra.run.smithi039.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 290 kB/s | 43 kB 00:00 2024-06-03T05:57:04.620 INFO:teuthology.orchestra.run.smithi039.stdout:(19/36): ceph-osd-19.0.0-2946.g3d89accf.el9.x86 15 MB/s | 17 MB 00:01 2024-06-03T05:57:04.637 INFO:teuthology.orchestra.run.smithi039.stdout:(20/36): luarocks-3.9.2-5.el9.noarch.rpm 419 kB/s | 151 kB 00:00 2024-06-03T05:57:04.662 INFO:teuthology.orchestra.run.smithi039.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 822 kB/s | 172 kB 00:00 2024-06-03T05:57:04.696 INFO:teuthology.orchestra.run.smithi039.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 182 kB/s | 11 kB 00:00 2024-06-03T05:57:04.721 INFO:teuthology.orchestra.run.smithi039.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2024-06-03T05:57:04.755 INFO:teuthology.orchestra.run.smithi039.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 397 kB/s | 23 kB 00:00 2024-06-03T05:57:04.780 INFO:teuthology.orchestra.run.smithi039.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 331 kB/s | 19 kB 00:00 2024-06-03T05:57:04.814 INFO:teuthology.orchestra.run.smithi039.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 336 kB/s | 20 kB 00:00 2024-06-03T05:57:04.848 INFO:teuthology.orchestra.run.smithi039.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 693 kB/s | 46 kB 00:00 2024-06-03T05:57:04.898 INFO:teuthology.orchestra.run.smithi039.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 942 kB/s | 79 kB 00:00 2024-06-03T05:57:04.923 INFO:teuthology.orchestra.run.smithi039.stdout:(29/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-06-03T05:57:04.949 INFO:teuthology.orchestra.run.smithi039.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-06-03T05:57:04.965 INFO:teuthology.orchestra.run.smithi039.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 245 kB/s | 16 kB 00:00 2024-06-03T05:57:04.991 INFO:teuthology.orchestra.run.smithi039.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-06-03T05:57:05.008 INFO:teuthology.orchestra.run.smithi039.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 609 kB/s | 36 kB 00:00 2024-06-03T05:57:05.058 INFO:teuthology.orchestra.run.smithi039.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-06-03T05:57:05.083 INFO:teuthology.orchestra.run.smithi039.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-06-03T05:57:05.108 INFO:teuthology.orchestra.run.smithi039.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-06-03T05:57:05.110 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:57:05.110 INFO:teuthology.orchestra.run.smithi039.stdout:Total 12 MB/s | 30 MB 00:02 2024-06-03T05:57:05.258 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:57:05.302 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:57:05.302 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:57:05.748 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:57:05.749 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:57:07.190 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:57:07.245 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-06-03T05:57:07.284 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-06-03T05:57:07.314 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-06-03T05:57:07.389 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-06-03T05:57:07.447 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/36 2024-06-03T05:57:07.479 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-06-03T05:57:07.576 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-06-03T05:57:07.661 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-06-03T05:57:07.708 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-06-03T05:57:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-06-03T05:57:07.772 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-06-03T05:57:07.807 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-06-03T05:57:07.840 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-06-03T05:57:07.881 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-06-03T05:57:07.925 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-06-03T05:57:07.959 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-06-03T05:57:07.997 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-06-03T05:57:08.074 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-06-03T05:57:08.227 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-06-03T05:57:08.318 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-06-03T05:57:08.376 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-06-03T05:57:08.414 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-06-03T05:57:08.473 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-06-03T05:57:08.537 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-06-03T05:57:08.736 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-06-03T05:57:08.785 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-06-03T05:57:08.867 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-06-03T05:57:08.965 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-06-03T05:57:09.001 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-06-03T05:57:09.126 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-06-03T05:57:09.300 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9. 31/36 2024-06-03T05:57:09.327 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 32/36 2024-06-03T05:57:09.355 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 32/36 2024-06-03T05:57:09.355 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:57:09.355 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-03T05:57:09.355 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-03T05:57:09.356 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-03T05:57:09.356 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:10.879 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 33/36 2024-06-03T05:57:10.908 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 33/36 2024-06-03T05:57:10.908 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:57:10.908 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-03T05:57:10.908 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-03T05:57:10.909 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-03T05:57:10.909 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:11.380 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 34/36 2024-06-03T05:57:11.405 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 34/36 2024-06-03T05:57:11.405 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:57:11.406 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-03T05:57:11.406 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-03T05:57:11.406 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-03T05:57:11.406 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:11.647 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 35/36 2024-06-03T05:57:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 35/36 2024-06-03T05:57:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:57:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-03T05:57:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-03T05:57:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-03T05:57:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:11.738 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 36/36 2024-06-03T05:57:13.442 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 36/36 2024-06-03T05:57:13.442 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 1/36 2024-06-03T05:57:13.442 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 2/36 2024-06-03T05:57:13.442 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 3/36 2024-06-03T05:57:13.442 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 4/36 2024-06-03T05:57:13.442 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 5/36 2024-06-03T05:57:13.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9. 6/36 2024-06-03T05:57:13.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-06-03T05:57:13.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-06-03T05:57:13.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-06-03T05:57:13.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-06-03T05:57:13.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-06-03T05:57:13.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-06-03T05:57:13.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-06-03T05:57:13.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-06-03T05:57:13.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-06-03T05:57:13.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-06-03T05:57:13.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-06-03T05:57:13.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-06-03T05:57:13.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 19/36 2024-06-03T05:57:13.444 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-06-03T05:57:13.444 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-06-03T05:57:13.444 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-06-03T05:57:13.444 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-06-03T05:57:13.444 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-06-03T05:57:13.444 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-06-03T05:57:13.444 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-06-03T05:57:13.444 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-06-03T05:57:13.444 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-06-03T05:57:13.444 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-06-03T05:57:13.444 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-06-03T05:57:13.444 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-06-03T05:57:13.444 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-06-03T05:57:13.444 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-06-03T05:57:13.445 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-06-03T05:57:13.445 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-06-03T05:57:14.368 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-06-03T05:57:14.368 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:14.368 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:57:14.368 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:57:14.368 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:57:14.368 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:57:14.368 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:57:14.369 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:57:14.369 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:57:14.369 INFO:teuthology.orchestra.run.smithi039.stdout: lua-5.4.4-4.el9.x86_64 2024-06-03T05:57:14.369 INFO:teuthology.orchestra.run.smithi039.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-03T05:57:14.369 INFO:teuthology.orchestra.run.smithi039.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-03T05:57:14.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-03T05:57:14.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-03T05:57:14.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-03T05:57:14.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-03T05:57:14.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-03T05:57:14.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-03T05:57:14.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-03T05:57:14.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-03T05:57:14.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-03T05:57:14.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-03T05:57:14.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-03T05:57:14.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-03T05:57:14.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-03T05:57:14.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-03T05:57:14.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-03T05:57:14.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply-3.11-14.el9.noarch 2024-06-03T05:57:14.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-03T05:57:14.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-03T05:57:14.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-03T05:57:14.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-03T05:57:14.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-03T05:57:14.371 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-03T05:57:14.371 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-03T05:57:14.371 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-03T05:57:14.371 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-03T05:57:14.371 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-03T05:57:14.371 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-03T05:57:14.371 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:14.371 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:57:14.696 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-base 2024-06-03T05:57:15.309 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:56 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:57:15.394 INFO:teuthology.orchestra.run.smithi039.stdout:Package ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:57:15.437 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:57:15.438 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-06-03T05:57:15.438 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:57:15.491 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install cephadm 2024-06-03T05:57:16.093 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:57 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:57:16.222 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:57:16.223 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:16.223 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-06-03T05:57:16.223 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:16.223 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:57:16.223 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 763 k 2024-06-03T05:57:16.223 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:16.224 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:57:16.224 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:16.224 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-06-03T05:57:16.224 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:16.224 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 763 k 2024-06-03T05:57:16.224 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 769 k 2024-06-03T05:57:16.224 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:57:16.656 INFO:teuthology.orchestra.run.smithi039.stdout:cephadm-19.0.0-2946.g3d89accf.el9.noarch.rpm 1.7 MB/s | 763 kB 00:00 2024-06-03T05:57:16.657 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:57:16.657 INFO:teuthology.orchestra.run.smithi039.stdout:Total 1.7 MB/s | 763 kB 00:00 2024-06-03T05:57:16.657 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:57:16.663 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:57:16.663 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:57:16.669 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:57:16.669 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:57:16.755 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:57:17.219 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 1/1 2024-06-03T05:57:17.313 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 1/1 2024-06-03T05:57:17.704 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 1/1 2024-06-03T05:57:18.124 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 1/1 2024-06-03T05:57:18.124 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:18.124 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:57:18.125 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:57:18.125 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:18.125 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:57:18.341 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-immutable-object-cache 2024-06-03T05:57:18.945 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:59 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:57:19.073 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:57:19.074 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:19.074 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2024-06-03T05:57:19.074 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:19.074 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:57:19.074 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 145 k 2024-06-03T05:57:19.074 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:19.074 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:57:19.074 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:19.074 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-06-03T05:57:19.074 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:19.075 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 145 k 2024-06-03T05:57:19.075 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 435 k 2024-06-03T05:57:19.075 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:57:19.307 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-immutable-object-cache-19.0.0-2946.g3d89ac 629 kB/s | 145 kB 00:00 2024-06-03T05:57:19.307 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:57:19.307 INFO:teuthology.orchestra.run.smithi039.stdout:Total 625 kB/s | 145 kB 00:00 2024-06-03T05:57:19.307 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:57:19.314 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:57:19.315 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:57:19.374 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:57:19.375 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:57:19.530 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:57:19.623 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf. 1/1 2024-06-03T05:57:19.649 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf. 1/1 2024-06-03T05:57:19.649 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:57:19.649 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-03T05:57:19.649 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:20.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf. 1/1 2024-06-03T05:57:20.544 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:20.544 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:57:20.544 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:57:20.544 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:20.545 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:57:20.752 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr 2024-06-03T05:57:21.350 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:02 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:57:21.436 INFO:teuthology.orchestra.run.smithi039.stdout:Package ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:57:21.481 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:57:21.482 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-06-03T05:57:21.482 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:57:21.545 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-dashboard 2024-06-03T05:57:22.144 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:03 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:57:22.270 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:57:22.271 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:22.272 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-06-03T05:57:22.272 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:22.272 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:57:22.272 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 3.5 M 2024-06-03T05:57:22.272 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-06-03T05:57:22.272 INFO:teuthology.orchestra.run.smithi039.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-03T05:57:22.272 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 32 k 2024-06-03T05:57:22.272 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 16 k 2024-06-03T05:57:22.272 INFO:teuthology.orchestra.run.smithi039.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-03T05:57:22.272 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-03T05:57:22.272 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-03T05:57:22.272 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-03T05:57:22.273 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-03T05:57:22.273 INFO:teuthology.orchestra.run.smithi039.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-03T05:57:22.273 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-03T05:57:22.273 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-03T05:57:22.273 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:22.273 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:57:22.273 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:22.273 INFO:teuthology.orchestra.run.smithi039.stdout:Install 12 Packages 2024-06-03T05:57:22.273 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:22.274 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 8.6 M 2024-06-03T05:57:22.274 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 103 M 2024-06-03T05:57:22.274 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:57:22.935 INFO:teuthology.orchestra.run.smithi039.stdout:(1/12): ceph-prometheus-alerts-19.0.0-2946.g3d8 83 kB/s | 16 kB 00:00 2024-06-03T05:57:23.035 INFO:teuthology.orchestra.run.smithi039.stdout:(2/12): ceph-grafana-dashboards-19.0.0-2946.g3d 106 kB/s | 32 kB 00:00 2024-06-03T05:57:23.111 INFO:teuthology.orchestra.run.smithi039.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-06-03T05:57:23.169 INFO:teuthology.orchestra.run.smithi039.stdout:(4/12): ceph-mgr-dashboard-19.0.0-2946.g3d89acc 8.1 MB/s | 3.5 MB 00:00 2024-06-03T05:57:23.195 INFO:teuthology.orchestra.run.smithi039.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 269 kB 00:00 2024-06-03T05:57:23.370 INFO:teuthology.orchestra.run.smithi039.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.3 MB/s | 863 kB 00:00 2024-06-03T05:57:23.412 INFO:teuthology.orchestra.run.smithi039.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 89 kB/s | 19 kB 00:00 2024-06-03T05:57:23.537 INFO:teuthology.orchestra.run.smithi039.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-06-03T05:57:23.588 INFO:teuthology.orchestra.run.smithi039.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 613 kB/s | 31 kB 00:00 2024-06-03T05:57:23.613 INFO:teuthology.orchestra.run.smithi039.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 822 kB/s | 144 kB 00:00 2024-06-03T05:57:23.655 INFO:teuthology.orchestra.run.smithi039.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-06-03T05:57:23.788 INFO:teuthology.orchestra.run.smithi039.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.9 MB/s | 2.0 MB 00:00 2024-06-03T05:57:23.792 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:57:23.792 INFO:teuthology.orchestra.run.smithi039.stdout:Total 5.7 MB/s | 8.6 MB 00:01 2024-06-03T05:57:23.841 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:57:23.859 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:57:23.859 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:57:24.050 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:57:24.050 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:57:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:57:24.665 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-06-03T05:57:24.700 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-06-03T05:57:24.747 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-06-03T05:57:24.776 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-06-03T05:57:24.883 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-06-03T05:57:25.028 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-06-03T05:57:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-06-03T05:57:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-06-03T05:57:25.210 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-06-03T05:57:25.243 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9 10/12 2024-06-03T05:57:26.348 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el 11/12 2024-06-03T05:57:26.384 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noa 12/12 2024-06-03T05:57:27.707 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noa 12/12 2024-06-03T05:57:27.707 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el 1/12 2024-06-03T05:57:27.707 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noa 2/12 2024-06-03T05:57:27.707 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9 3/12 2024-06-03T05:57:27.707 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-06-03T05:57:27.708 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-06-03T05:57:27.708 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-06-03T05:57:27.708 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-06-03T05:57:27.708 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-06-03T05:57:27.708 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-06-03T05:57:27.708 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-06-03T05:57:27.708 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-06-03T05:57:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-06-03T05:57:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:28.026 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:57:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-03T05:57:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:57:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:57:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:57:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-03T05:57:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-03T05:57:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-03T05:57:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-03T05:57:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-03T05:57:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-03T05:57:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-03T05:57:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-03T05:57:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:28.027 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:57:28.256 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-03T05:57:28.853 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:09 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:57:28.981 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:57:28.982 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================= 2024-06-03T05:57:28.982 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-06-03T05:57:28.982 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================= 2024-06-03T05:57:28.982 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:57:28.982 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 7.4 M 2024-06-03T05:57:28.983 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-06-03T05:57:28.983 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-03T05:57:28.983 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-03T05:57:28.983 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-03T05:57:28.983 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-03T05:57:28.983 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-03T05:57:28.983 INFO:teuthology.orchestra.run.smithi039.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-03T05:57:28.983 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-03T05:57:28.983 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-03T05:57:28.983 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-03T05:57:28.983 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-03T05:57:28.983 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-03T05:57:28.983 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:28.983 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:57:28.984 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================= 2024-06-03T05:57:28.984 INFO:teuthology.orchestra.run.smithi039.stdout:Install 12 Packages 2024-06-03T05:57:28.984 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:28.984 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 38 M 2024-06-03T05:57:28.984 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 209 M 2024-06-03T05:57:28.984 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:57:29.289 INFO:teuthology.orchestra.run.smithi039.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-06-03T05:57:29.356 INFO:teuthology.orchestra.run.smithi039.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 491 kB/s | 33 kB 00:00 2024-06-03T05:57:29.381 INFO:teuthology.orchestra.run.smithi039.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-06-03T05:57:29.448 INFO:teuthology.orchestra.run.smithi039.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 273 kB/s | 18 kB 00:00 2024-06-03T05:57:29.541 INFO:teuthology.orchestra.run.smithi039.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 16 MB/s | 3.0 MB 00:00 2024-06-03T05:57:29.566 INFO:teuthology.orchestra.run.smithi039.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 301 kB/s | 35 kB 00:00 2024-06-03T05:57:29.674 INFO:teuthology.orchestra.run.smithi039.stdout:(7/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 35 MB/s | 4.7 MB 00:00 2024-06-03T05:57:29.699 INFO:teuthology.orchestra.run.smithi039.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.8 MB/s | 244 kB 00:00 2024-06-03T05:57:29.800 INFO:teuthology.orchestra.run.smithi039.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-29 11 MB/s | 7.4 MB 00:00 2024-06-03T05:57:29.825 INFO:teuthology.orchestra.run.smithi039.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.0 MB/s | 250 kB 00:00 2024-06-03T05:57:29.900 INFO:teuthology.orchestra.run.smithi039.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-06-03T05:57:30.250 INFO:teuthology.orchestra.run.smithi039.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-06-03T05:57:30.251 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:57:30.252 INFO:teuthology.orchestra.run.smithi039.stdout:Total 30 MB/s | 38 MB 00:01 2024-06-03T05:57:30.615 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:57:30.644 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:57:30.644 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:57:31.116 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:57:31.117 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:57:31.897 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:57:32.091 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-03T05:57:32.141 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-03T05:57:32.253 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-03T05:57:32.293 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-03T05:57:32.479 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-03T05:57:32.508 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-03T05:57:32.597 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-03T05:57:33.281 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-03T05:57:33.409 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-03T05:57:34.733 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-03T05:57:35.835 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-03T05:57:35.866 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89a 12/12 2024-06-03T05:57:38.291 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89a 12/12 2024-06-03T05:57:38.291 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89a 1/12 2024-06-03T05:57:38.291 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-03T05:57:38.291 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-03T05:57:38.291 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-03T05:57:38.291 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-03T05:57:38.292 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-03T05:57:38.292 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-03T05:57:38.292 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-03T05:57:38.292 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-03T05:57:38.292 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-03T05:57:38.292 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-03T05:57:38.786 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-03T05:57:38.786 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:38.786 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:57:38.787 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:57:38.787 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-03T05:57:38.787 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-03T05:57:38.787 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-03T05:57:38.787 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-03T05:57:38.787 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-03T05:57:38.787 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-03T05:57:38.787 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-03T05:57:38.787 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-03T05:57:38.787 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-03T05:57:38.787 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-03T05:57:38.787 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-03T05:57:38.787 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:38.788 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:57:39.187 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-rook 2024-06-03T05:57:39.794 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:20 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:57:39.923 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:57:39.924 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:39.924 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-06-03T05:57:39.924 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:39.925 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:57:39.925 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 49 k 2024-06-03T05:57:39.925 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-06-03T05:57:39.925 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-03T05:57:39.925 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-03T05:57:39.925 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-03T05:57:39.925 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-03T05:57:39.925 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-03T05:57:39.926 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-03T05:57:39.926 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-03T05:57:39.926 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-03T05:57:39.926 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-03T05:57:39.926 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-03T05:57:39.926 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-03T05:57:39.927 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-03T05:57:39.927 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:39.927 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:57:39.927 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:39.927 INFO:teuthology.orchestra.run.smithi039.stdout:Install 13 Packages 2024-06-03T05:57:39.927 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:39.928 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 2.2 M 2024-06-03T05:57:39.928 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 26 M 2024-06-03T05:57:39.928 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:57:40.497 INFO:teuthology.orchestra.run.smithi039.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 317 kB/s | 26 kB 00:00 2024-06-03T05:57:40.514 INFO:teuthology.orchestra.run.smithi039.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 196 kB/s | 19 kB 00:00 2024-06-03T05:57:40.597 INFO:teuthology.orchestra.run.smithi039.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 222 kB 00:00 2024-06-03T05:57:40.622 INFO:teuthology.orchestra.run.smithi039.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-06-03T05:57:40.639 INFO:teuthology.orchestra.run.smithi039.stdout:(5/13): ceph-mgr-rook-19.0.0-2946.g3d89accf.el9 220 kB/s | 49 kB 00:00 2024-06-03T05:57:40.664 INFO:teuthology.orchestra.run.smithi039.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.3 MB/s | 54 kB 00:00 2024-06-03T05:57:40.690 INFO:teuthology.orchestra.run.smithi039.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 279 kB 00:00 2024-06-03T05:57:40.916 INFO:teuthology.orchestra.run.smithi039.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 117 kB/s | 32 kB 00:00 2024-06-03T05:57:40.932 INFO:teuthology.orchestra.run.smithi039.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-06-03T05:57:41.074 INFO:teuthology.orchestra.run.smithi039.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 419 kB/s | 59 kB 00:00 2024-06-03T05:57:41.199 INFO:teuthology.orchestra.run.smithi039.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 718 kB/s | 90 kB 00:00 2024-06-03T05:57:41.225 INFO:teuthology.orchestra.run.smithi039.stdout:(12/13): python3-google-auth-2.29.0-1.el9.noarc 385 kB/s | 206 kB 00:00 2024-06-03T05:57:41.258 INFO:teuthology.orchestra.run.smithi039.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2024-06-03T05:57:41.261 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:57:41.261 INFO:teuthology.orchestra.run.smithi039.stdout:Total 1.6 MB/s | 2.2 MB 00:01 2024-06-03T05:57:41.331 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:57:41.346 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:57:41.347 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:57:41.444 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:57:41.445 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:57:42.090 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:57:42.277 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-03T05:57:42.337 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-03T05:57:42.393 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-03T05:57:42.428 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-03T05:57:42.470 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-03T05:57:42.569 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-03T05:57:42.687 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-06-03T05:57:42.739 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-03T05:57:43.525 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-03T05:57:43.576 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-03T05:57:43.610 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-03T05:57:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-03T05:57:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 13/13 2024-06-03T05:57:44.425 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 13/13 2024-06-03T05:57:44.425 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 1/13 2024-06-03T05:57:44.425 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-03T05:57:44.425 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-03T05:57:44.425 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-03T05:57:44.425 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-03T05:57:44.425 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-03T05:57:44.425 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-03T05:57:44.425 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-03T05:57:44.425 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-03T05:57:44.425 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-06-03T05:57:44.425 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-03T05:57:44.425 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-03T05:57:44.740 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-03T05:57:44.740 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:44.740 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:57:44.741 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:57:44.741 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-03T05:57:44.741 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-03T05:57:44.741 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-03T05:57:44.741 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-03T05:57:44.741 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-03T05:57:44.741 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-03T05:57:44.741 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-03T05:57:44.741 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-03T05:57:44.741 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-03T05:57:44.741 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-03T05:57:44.741 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-03T05:57:44.741 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-03T05:57:44.742 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:44.742 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:57:44.995 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-cephadm 2024-06-03T05:57:45.594 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:26 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:57:45.723 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:57:45.724 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:45.724 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-06-03T05:57:45.724 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:45.724 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:57:45.724 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 142 k 2024-06-03T05:57:45.724 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-06-03T05:57:45.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-03T05:57:45.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-03T05:57:45.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-03T05:57:45.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-03T05:57:45.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-03T05:57:45.725 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:45.725 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:57:45.725 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:45.725 INFO:teuthology.orchestra.run.smithi039.stdout:Install 6 Packages 2024-06-03T05:57:45.725 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:45.726 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 7.0 M 2024-06-03T05:57:45.726 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 33 M 2024-06-03T05:57:45.726 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:57:46.110 INFO:teuthology.orchestra.run.smithi039.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-06-03T05:57:46.169 INFO:teuthology.orchestra.run.smithi039.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2946.g3d89accf.e 627 kB/s | 142 kB 00:00 2024-06-03T05:57:46.286 INFO:teuthology.orchestra.run.smithi039.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-06-03T05:57:46.545 INFO:teuthology.orchestra.run.smithi039.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 154 kB/s | 58 kB 00:00 2024-06-03T05:57:46.595 INFO:teuthology.orchestra.run.smithi039.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 165 kB/s | 51 kB 00:00 2024-06-03T05:57:46.703 INFO:teuthology.orchestra.run.smithi039.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 925 kB/s | 548 kB 00:00 2024-06-03T05:57:46.704 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:57:46.704 INFO:teuthology.orchestra.run.smithi039.stdout:Total 7.1 MB/s | 7.0 MB 00:00 2024-06-03T05:57:46.779 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:57:46.791 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:57:46.791 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:57:46.896 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:57:46.896 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:57:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:57:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-03T05:57:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-03T05:57:48.151 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-03T05:57:48.236 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-03T05:57:48.307 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-03T05:57:48.334 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 6/6 2024-06-03T05:57:49.196 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 6/6 2024-06-03T05:57:49.197 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 1/6 2024-06-03T05:57:49.197 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-03T05:57:49.197 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-03T05:57:49.197 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-03T05:57:49.197 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-03T05:57:49.463 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-03T05:57:49.463 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:49.463 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:57:49.463 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:57:49.464 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-03T05:57:49.464 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-03T05:57:49.464 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-03T05:57:49.464 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-03T05:57:49.464 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-03T05:57:49.464 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:49.465 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:57:49.687 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-fuse 2024-06-03T05:57:50.296 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:31 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:57:50.425 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:57:50.425 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:50.425 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-06-03T05:57:50.426 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:50.426 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:57:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 890 k 2024-06-03T05:57:50.426 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-06-03T05:57:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-03T05:57:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:50.426 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:57:50.426 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:50.426 INFO:teuthology.orchestra.run.smithi039.stdout:Install 2 Packages 2024-06-03T05:57:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:50.427 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 970 k 2024-06-03T05:57:50.427 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 2.8 M 2024-06-03T05:57:50.427 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:57:50.655 INFO:teuthology.orchestra.run.smithi039.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 756 kB/s | 80 kB 00:00 2024-06-03T05:57:50.830 INFO:teuthology.orchestra.run.smithi039.stdout:(2/2): ceph-fuse-19.0.0-2946.g3d89accf.el9.x86_ 3.1 MB/s | 890 kB 00:00 2024-06-03T05:57:50.831 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:57:50.831 INFO:teuthology.orchestra.run.smithi039.stdout:Total 2.3 MB/s | 970 kB 00:00 2024-06-03T05:57:50.843 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:57:50.867 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:57:50.867 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:57:50.950 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:57:50.951 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:57:51.142 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:57:51.303 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-03T05:57:51.374 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2/2 2024-06-03T05:57:52.010 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2/2 2024-06-03T05:57:52.011 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 1/2 2024-06-03T05:57:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-03T05:57:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:52.301 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:57:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-03T05:57:52.302 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:52.302 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:57:52.563 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-volume 2024-06-03T05:57:53.171 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:34 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:57:53.301 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:57:53.301 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:53.302 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-06-03T05:57:53.302 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:53.302 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:57:53.302 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 264 k 2024-06-03T05:57:53.302 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-06-03T05:57:53.302 INFO:teuthology.orchestra.run.smithi039.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-03T05:57:53.302 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:53.302 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:57:53.302 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:53.302 INFO:teuthology.orchestra.run.smithi039.stdout:Install 2 Packages 2024-06-03T05:57:53.302 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:53.303 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 341 k 2024-06-03T05:57:53.303 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 1.5 M 2024-06-03T05:57:53.303 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:57:53.669 INFO:teuthology.orchestra.run.smithi039.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 818 kB/s | 77 kB 00:00 2024-06-03T05:57:53.836 INFO:teuthology.orchestra.run.smithi039.stdout:(2/2): ceph-volume-19.0.0-2946.g3d89accf.el9.no 1.0 MB/s | 264 kB 00:00 2024-06-03T05:57:53.836 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:57:53.836 INFO:teuthology.orchestra.run.smithi039.stdout:Total 638 kB/s | 341 kB 00:00 2024-06-03T05:57:53.848 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:57:53.858 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:57:53.858 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:57:53.895 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:57:53.895 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:57:54.042 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:57:54.227 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-06-03T05:57:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2/2 2024-06-03T05:57:54.274 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2/2 2024-06-03T05:57:54.274 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:57:54.274 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-03T05:57:54.274 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:54.997 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 1/2 2024-06-03T05:57:55.290 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-06-03T05:57:55.290 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:55.290 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:57:55.290 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:57:55.290 INFO:teuthology.orchestra.run.smithi039.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-03T05:57:55.290 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:55.291 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:57:55.582 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librados-devel 2024-06-03T05:57:56.195 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:37 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:57:56.326 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:57:56.327 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:56.327 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2024-06-03T05:57:56.327 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:56.327 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:57:56.327 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 127 k 2024-06-03T05:57:56.327 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:56.327 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:57:56.327 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:56.328 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-06-03T05:57:56.328 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:56.328 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 127 k 2024-06-03T05:57:56.328 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 456 k 2024-06-03T05:57:56.328 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:57:56.558 INFO:teuthology.orchestra.run.smithi039.stdout:librados-devel-19.0.0-2946.g3d89accf.el9.x86_64 545 kB/s | 127 kB 00:00 2024-06-03T05:57:56.559 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:57:56.559 INFO:teuthology.orchestra.run.smithi039.stdout:Total 542 kB/s | 127 kB 00:00 2024-06-03T05:57:56.559 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:57:56.565 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:57:56.565 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:57:56.633 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:57:56.633 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:57:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:57:56.892 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:57:57.336 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:57:57.579 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:57:57.580 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:57.580 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:57:57.580 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:57:57.580 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:57.580 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:57:57.855 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install libcephfs2 2024-06-03T05:57:58.467 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:39 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:57:58.551 INFO:teuthology.orchestra.run.smithi039.stdout:Package libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:57:58.597 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:57:58.598 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-06-03T05:57:58.598 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:57:58.665 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install libcephfs-devel 2024-06-03T05:57:59.265 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:40 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:57:59.396 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:57:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:59.397 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2024-06-03T05:57:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:57:59.397 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 32 k 2024-06-03T05:57:59.397 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:57:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:57:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-06-03T05:57:59.397 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:57:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 32 k 2024-06-03T05:57:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 142 k 2024-06-03T05:57:59.398 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:57:59.636 INFO:teuthology.orchestra.run.smithi039.stdout:libcephfs-devel-19.0.0-2946.g3d89accf.el9.x86_6 133 kB/s | 32 kB 00:00 2024-06-03T05:57:59.637 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:57:59.637 INFO:teuthology.orchestra.run.smithi039.stdout:Total 132 kB/s | 32 kB 00:00 2024-06-03T05:57:59.637 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:57:59.641 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:57:59.641 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:57:59.660 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:57:59.661 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:57:59.743 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:57:59.850 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:58:00.215 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:58:00.435 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:58:00.436 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:58:00.436 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:58:00.436 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:58:00.436 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:58:00.436 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:58:00.598 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librados2 2024-06-03T05:58:01.197 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:42 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:58:01.281 INFO:teuthology.orchestra.run.smithi039.stdout:Package librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:58:01.325 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:58:01.326 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-06-03T05:58:01.327 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:58:01.402 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librbd1 2024-06-03T05:58:02.004 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:43 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:58:02.088 INFO:teuthology.orchestra.run.smithi039.stdout:Package librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:58:02.133 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:58:02.134 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-06-03T05:58:02.134 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:58:02.219 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rados 2024-06-03T05:58:02.824 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:43 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:58:02.908 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:58:02.953 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:58:02.954 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-06-03T05:58:02.954 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:58:03.026 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rgw 2024-06-03T05:58:03.629 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:44 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:58:03.714 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:58:03.759 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:58:03.760 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-06-03T05:58:03.760 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:58:03.830 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-cephfs 2024-06-03T05:58:04.437 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:45 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:58:04.521 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:58:04.567 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:58:04.568 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-06-03T05:58:04.568 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:58:04.640 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rbd 2024-06-03T05:58:05.244 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:46 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:58:05.328 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:58:05.372 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:58:05.373 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-06-03T05:58:05.373 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:58:05.443 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-fuse 2024-06-03T05:58:06.039 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:47 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:58:06.165 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:58:06.166 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:58:06.166 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-06-03T05:58:06.166 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:58:06.166 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:58:06.166 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 85 k 2024-06-03T05:58:06.167 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:58:06.167 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:58:06.167 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:58:06.167 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-06-03T05:58:06.167 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:58:06.167 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 85 k 2024-06-03T05:58:06.167 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 231 k 2024-06-03T05:58:06.167 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:58:06.353 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-fuse-19.0.0-2946.g3d89accf.el9.x86_64.rpm 459 kB/s | 85 kB 00:00 2024-06-03T05:58:06.353 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:58:06.353 INFO:teuthology.orchestra.run.smithi039.stdout:Total 456 kB/s | 85 kB 00:00 2024-06-03T05:58:06.353 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:58:06.361 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:58:06.361 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:58:06.427 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:58:06.427 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:58:06.575 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:58:06.686 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:58:07.052 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:58:07.303 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:58:07.304 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:58:07.304 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:58:07.304 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:58:07.304 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:58:07.304 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:58:07.569 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-mirror 2024-06-03T05:58:08.176 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:49 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:58:08.307 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:58:08.307 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:58:08.307 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2024-06-03T05:58:08.308 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:58:08.308 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:58:08.308 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-03T05:58:08.308 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:58:08.308 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:58:08.308 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:58:08.308 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-06-03T05:58:08.308 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:58:08.308 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 3.1 M 2024-06-03T05:58:08.308 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 13 M 2024-06-03T05:58:08.308 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:58:08.852 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-mirror-19.0.0-2946.g3d89accf.el9.x86_64.rpm 5.7 MB/s | 3.1 MB 00:00 2024-06-03T05:58:08.852 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:58:08.853 INFO:teuthology.orchestra.run.smithi039.stdout:Total 5.7 MB/s | 3.1 MB 00:00 2024-06-03T05:58:08.853 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:58:08.863 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:58:08.863 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:58:08.949 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:58:08.949 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:58:09.413 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:58:09.478 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:58:09.505 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:58:09.505 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:58:09.505 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-03T05:58:09.505 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-03T05:58:09.505 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-03T05:58:09.506 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:58:10.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:58:10.720 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:58:10.721 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:58:10.721 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:58:10.721 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:58:10.721 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:58:10.965 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-nbd 2024-06-03T05:58:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:52 ago on Mon 03 Jun 2024 05:56:19 AM UTC. 2024-06-03T05:58:11.699 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-06-03T05:58:11.700 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:58:11.700 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-06-03T05:58:11.700 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:58:11.700 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-06-03T05:58:11.700 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 171 k 2024-06-03T05:58:11.700 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:58:11.700 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-06-03T05:58:11.700 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-06-03T05:58:11.700 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-06-03T05:58:11.700 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:58:11.701 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 171 k 2024-06-03T05:58:11.701 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 494 k 2024-06-03T05:58:11.701 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-06-03T05:58:12.049 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-nbd-19.0.0-2946.g3d89accf.el9.x86_64.rpm 491 kB/s | 171 kB 00:00 2024-06-03T05:58:12.049 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:58:12.049 INFO:teuthology.orchestra.run.smithi039.stdout:Total 489 kB/s | 171 kB 00:00 2024-06-03T05:58:12.049 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-06-03T05:58:12.058 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-06-03T05:58:12.058 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-06-03T05:58:12.125 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-06-03T05:58:12.125 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-06-03T05:58:12.292 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-06-03T05:58:12.382 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:58:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:58:13.020 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:58:13.020 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:58:13.020 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-06-03T05:58:13.020 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:58:13.020 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-06-03T05:58:13.020 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-06-03T05:58:13.262 DEBUG:teuthology.parallel:result is None 2024-06-03T05:58:13.263 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-03T05:58:13.410 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-03T05:58:13.432 INFO:teuthology.orchestra.run.smithi039.stdout:19.0.0-2946.g3d89accf.el9 2024-06-03T05:58:13.432 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2946.g3d89accf.el9 2024-06-03T05:58:13.432 INFO:teuthology.task.install:The correct ceph version 19.0.0-2946.g3d89accf is installed. 2024-06-03T05:58:13.434 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-03T05:58:13.434 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-06-03T05:58:13.434 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-03T05:58:13.497 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-03T05:58:13.497 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-06-03T05:58:13.497 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2024-06-03T05:58:13.563 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-03T05:58:13.629 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-03T05:58:13.629 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-06-03T05:58:13.629 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-03T05:58:13.694 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-03T05:58:13.759 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-03T05:58:13.759 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-06-03T05:58:13.759 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2024-06-03T05:58:13.824 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-03T05:58:13.889 INFO:teuthology.run_tasks:Running task workunit... 2024-06-03T05:58:13.902 INFO:tasks.workunit:Pulling workunits from ref dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 2024-06-03T05:58:13.902 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-06-03T05:58:13.902 INFO:tasks.workunit:timeout=3h 2024-06-03T05:58:13.903 INFO:tasks.workunit:cleanup=True 2024-06-03T05:58:13.903 DEBUG:teuthology.orchestra.run.smithi039:> stat -- /home/ubuntu/cephtest/mnt.0 2024-06-03T05:58:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T05:58:13.945 INFO:teuthology.orchestra.run.smithi039.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-06-03T05:58:13.946 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-06-03T05:58:14.001 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-06-03T05:58:14.001 DEBUG:teuthology.orchestra.run.smithi039:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-06-03T05:58:14.059 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 2024-06-03T05:58:14.115 INFO:tasks.workunit.client.0.smithi039.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-06-03T05:59:30.235 INFO:tasks.workunit.client.0.smithi039.stderr:Updating files: 96% (12083/12565) Updating files: 97% (12189/12565) Updating files: 98% (12314/12565) Updating files: 99% (12440/12565) Updating files: 100% (12565/12565) Updating files: 100% (12565/12565), done. 2024-06-03T05:59:30.718 INFO:tasks.workunit.client.0.smithi039.stderr:Note: switching to 'dfcc4b3c2c55532c4c04fa47551c9df1dffdf746'. 2024-06-03T05:59:30.718 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-06-03T05:59:30.718 INFO:tasks.workunit.client.0.smithi039.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-06-03T05:59:30.718 INFO:tasks.workunit.client.0.smithi039.stderr:changes and commit them, and you can discard any commits you make in this 2024-06-03T05:59:30.718 INFO:tasks.workunit.client.0.smithi039.stderr:state without impacting any branches by switching back to a branch. 2024-06-03T05:59:30.719 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-06-03T05:59:30.719 INFO:tasks.workunit.client.0.smithi039.stderr:If you want to create a new branch to retain commits you create, you may 2024-06-03T05:59:30.719 INFO:tasks.workunit.client.0.smithi039.stderr:do so (now or later) by using -c with the switch command. Example: 2024-06-03T05:59:30.719 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-06-03T05:59:30.719 INFO:tasks.workunit.client.0.smithi039.stderr: git switch -c 2024-06-03T05:59:30.719 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-06-03T05:59:30.719 INFO:tasks.workunit.client.0.smithi039.stderr:Or undo this operation with: 2024-06-03T05:59:30.719 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-06-03T05:59:30.719 INFO:tasks.workunit.client.0.smithi039.stderr: git switch - 2024-06-03T05:59:30.719 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-06-03T05:59:30.719 INFO:tasks.workunit.client.0.smithi039.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-06-03T05:59:30.719 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-06-03T05:59:30.719 INFO:tasks.workunit.client.0.smithi039.stderr:HEAD is now at dfcc4b3c2c5 Merge pull request #57814 from zdover23/wip-doc-2024-06-01-backport-57804-to-squid 2024-06-03T05:59:30.723 DEBUG:teuthology.orchestra.run.smithi039:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-06-03T05:59:30.780 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-06-03T05:59:30.780 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-06-03T05:59:30.836 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-06-03T05:59:30.836 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-06-03T05:59:30.837 DEBUG:teuthology.orchestra.run.smithi039:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-06-03T05:59:30.906 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-03T05:59:30.912 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-03T05:59:30.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-06-03T05:59:30.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-06-03T05:59:30.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-06-03T05:59:30.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-06-03T05:59:30.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-06-03T05:59:30.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-06-03T05:59:30.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-06-03T05:59:30.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-06-03T05:59:30.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-mapper 2024-06-03T05:59:30.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-06-03T05:59:30.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-06-03T05:59:30.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-06-03T05:59:30.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-06-03T05:59:30.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-06-03T05:59:30.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-06-03T05:59:30.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=562d63df-9949-45e5-983b-009533c6801f --auth-supported=none ' 2024-06-03T05:59:30.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-06-03T05:59:30.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-06-03T05:59:30.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-06-03T05:59:30.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-03T05:59:30.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-06-03T05:59:30.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-06-03T05:59:30.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-06-03T05:59:30.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2024-06-03T05:59:30.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2024-06-03T05:59:30.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-06-03T05:59:30.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:59:30.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-06-03T05:59:30.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:59:30.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:59:30.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:59:30.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:59:30.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:59:30.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:59:30.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:59:30.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:59:30.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:59:30.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:59:30.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:59:30.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:59:30.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:59:30.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:59:30.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:59:30.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:59:30.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:59:30.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:59:30.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-06-03T05:59:30.929 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:59:30.929 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:59:30.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28700 2024-06-03T05:59:30.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28700 2024-06-03T05:59:30.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:59:30.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:59:30.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2024-06-03T05:59:30.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:59:30.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:59:30.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28700 2024-06-03T05:59:30.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.28700 2024-06-03T05:59:30.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:59:30.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-03T05:59:30.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-03T05:59:30.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:59:30.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-06-03T05:59:30.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-06-03T05:59:30.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-06-03T05:59:30.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-06-03T05:59:30.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-06-03T05:59:30.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-06-03T05:59:30.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-06-03T05:59:30.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-06-03T05:59:30.934 INFO:tasks.workunit.client.0.smithi039.stdout:Dir: td/osd-mapper 2024-06-03T05:59:30.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-06-03T05:59:30.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2024-06-03T05:59:30.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-06-03T05:59:30.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-06-03T05:59:30.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-06-03T05:59:30.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-06-03T05:59:30.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-06-03T05:59:30.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-06-03T05:59:30.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-06-03T05:59:30.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-06-03T05:59:30.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-06-03T05:59:32.635 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 344b9b28-d74e-4937-b935-279b9ccb0e9d 2024-06-03T05:59:32.893 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T05:59:32.970 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T05:59:32.970+0000 7fd4b512d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:59:32.971 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T05:59:32.972+0000 7fd4b512d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:59:32.972 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T05:59:32.973+0000 7fd4b512d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:59:32.973 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T05:59:32.973+0000 7fd4b512d640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-06-03T05:59:32.973 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T05:59:32.973+0000 7fd4b512d640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-06-03T05:59:32.973 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T05:59:32.973+0000 7fd4b512d640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-06-03T05:59:33.045 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T05:59:33.045+0000 7fd4b512d640 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-06-03T05:59:38.849 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T05:59:39.235 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T05:59:39.265 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T05:59:39.265+0000 7f849e144640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:59:39.266 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T05:59:39.267+0000 7f849e144640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:59:39.267 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T05:59:39.268+0000 7f849e144640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:59:39.560 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T05:59:39.789 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T05:59:39.789+0000 7f849e144640 -1 Falling back to public interface 2024-06-03T05:59:40.872 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T05:59:42.084 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T05:59:42.084+0000 7f849e144640 -1 osd.0 0 log_to_monitors true 2024-06-03T05:59:42.167 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T05:59:43.511 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T05:59:44.841 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T05:59:46.141 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T05:59:47.442 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T05:59:48.855 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T05:59:50.156 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T05:59:51.458 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T05:59:52.817 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T05:59:54.444 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T05:59:55.969 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T05:59:57.451 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T05:59:59.062 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:00:00.680 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:00:02.039 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:00:03.445 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:00:05.338 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:00:06.712 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:00:07.010 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1535839800,v1:127.0.0.1:6803/1535839800] [v2:127.0.0.1:6804/1535839800,v1:127.0.0.1:6805/1535839800] exists,up 344b9b28-d74e-4937-b935-279b9ccb0e9d 2024-06-03T06:00:07.013 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 e3b37cc4-be5d-4269-b774-fb638e2c898a 2024-06-03T06:00:07.402 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:00:07.446 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:07.446+0000 7f410bc01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:00:07.470 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:07.470+0000 7f410bc01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:00:07.471 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:07.471+0000 7f410bc01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:00:07.471 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:07.471+0000 7f410bc01640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-06-03T06:00:07.471 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:07.471+0000 7f410bc01640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-06-03T06:00:07.472 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:07.471+0000 7f410bc01640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-06-03T06:00:07.535 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:07.535+0000 7f410bc01640 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-06-03T06:00:13.632 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:00:14.008 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:00:14.037 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:14.037+0000 7f027212d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:00:14.039 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:14.039+0000 7f027212d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:00:14.040 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:14.040+0000 7f027212d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:00:14.293 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:00:14.559 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:14.559+0000 7f027212d640 -1 Falling back to public interface 2024-06-03T06:00:15.579 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:00:16.881 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:00:17.045 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:17.045+0000 7f027212d640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:00:18.254 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:00:19.554 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:00:20.852 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:00:22.150 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:00:23.454 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:00:24.846 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:00:26.159 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:00:27.489 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:00:28.872 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:00:30.369 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:00:32.167 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:00:33.665 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:00:35.345 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:00:36.695 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:00:38.356 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:00:39.793 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:00:40.089 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2138028059,v1:127.0.0.1:6811/2138028059] [v2:127.0.0.1:6812/2138028059,v1:127.0.0.1:6813/2138028059] exists,up e3b37cc4-be5d-4269-b774-fb638e2c898a 2024-06-03T06:00:40.093 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 9f46a95d-e269-4231-8cbe-9e240418bf27 2024-06-03T06:00:40.453 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:00:40.497 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:40.497+0000 7f49fff44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:00:40.538 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:40.538+0000 7f49fff44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:00:40.539 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:40.539+0000 7f49fff44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:00:40.539 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:40.539+0000 7f49fff44640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-06-03T06:00:40.539 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:40.539+0000 7f49fff44640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-06-03T06:00:40.539 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:40.539+0000 7f49fff44640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-06-03T06:00:40.613 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:40.613+0000 7f49fff44640 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-06-03T06:00:46.575 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T06:00:46.952 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:00:46.982 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:46.981+0000 7f66f7e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:00:46.984 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:46.983+0000 7f66f7e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:00:46.985 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:46.984+0000 7f66f7e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:00:47.240 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:00:47.504 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:47.503+0000 7f66f7e19640 -1 Falling back to public interface 2024-06-03T06:00:48.528 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:00:49.824 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:00:49.832 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:00:49.831+0000 7f66f7e19640 -1 osd.2 0 log_to_monitors true 2024-06-03T06:00:51.191 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:00:52.539 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:00:53.837 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:00:55.139 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:00:56.490 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:00:57.863 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:00:59.167 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:01:00.519 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:01:01.941 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:01:03.241 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:01:04.742 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:01:06.150 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:01:07.544 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:01:09.118 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:01:10.706 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:01:12.044 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:01:13.655 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:01:13.955 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2272727427,v1:127.0.0.1:6819/2272727427] [v2:127.0.0.1:6820/2272727427,v1:127.0.0.1:6821/2272727427] exists,up 9f46a95d-e269-4231-8cbe-9e240418bf27 2024-06-03T06:01:14.341 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' created 2024-06-03T06:01:16.279 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836499 2024-06-03T06:01:16.569 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672974 2024-06-03T06:01:16.863 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509444 2024-06-03T06:01:19.456 INFO:tasks.workunit.client.0.smithi039.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-06-03T06:01:19.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-06-03T06:01:19.555 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: { 2024-06-03T06:01:19.555 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-06-03T06:01:19.555 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:01:19.569 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: { 2024-06-03T06:01:19.569 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-06-03T06:01:19.569 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:01:19.583 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: { 2024-06-03T06:01:19.583 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-06-03T06:01:19.583 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:01:19.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-06-03T06:01:19.696 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: { 2024-06-03T06:01:19.696 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-06-03T06:01:19.696 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:01:19.710 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: { 2024-06-03T06:01:19.710 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-06-03T06:01:19.710 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:01:19.725 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: { 2024-06-03T06:01:19.725 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-06-03T06:01:19.725 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:01:19.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-06-03T06:01:19.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-06-03T06:01:19.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-06-03T06:01:19.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-06-03T06:01:19.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-06-03T06:01:19.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-06-03T06:01:19.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-06-03T06:01:20.065 INFO:tasks.workunit.client.0.smithi039.stdout:created pool test snap snap01 2024-06-03T06:01:20.358 INFO:tasks.workunit.client.0.smithi039.stdout:created pool test snap snap02 2024-06-03T06:01:20.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-06-03T06:01:20.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-06-03T06:01:20.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-06-03T06:01:20.885 INFO:tasks.workunit.client.0.smithi039.stdout:created pool test snap snap13 2024-06-03T06:01:20.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-06-03T06:01:20.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-06-03T06:01:20.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-06-03T06:01:21.306 INFO:tasks.workunit.client.0.smithi039.stdout:created pool test snap snap24 2024-06-03T06:01:21.514 INFO:tasks.workunit.client.0.smithi039.stdout:created pool test snap snap25 2024-06-03T06:01:21.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 12470 2024-06-03T06:01:21.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-06-03T06:01:21.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-06-03T06:01:21.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-06-03T06:01:22.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-06-03T06:01:22.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-06-03T06:01:22.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-06-03T06:01:22.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-06-03T06:01:22.343 INFO:tasks.workunit.client.0.smithi039.stdout:pgid is 1.3 (primary: osd.1) 2024-06-03T06:01:22.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-06-03T06:01:22.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-06-03T06:01:22.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local pgid=1.3 2024-06-03T06:01:22.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: ceph pg dump pgs_brief 2024-06-03T06:01:22.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-06-03T06:01:22.613 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs_brief 2024-06-03T06:01:22.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: local prim_osd=1 2024-06-03T06:01:22.624 INFO:tasks.workunit.client.0.smithi039.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-06-03T06:01:22.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-06-03T06:01:22.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: get_asok_path osd.1 2024-06-03T06:01:22.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-03T06:01:22.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-03T06:01:22.626 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:01:22.626 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:01:22.626 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28700 2024-06-03T06:01:22.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.28700/ceph-osd.1.asok 2024-06-03T06:01:22.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: CEPH_ARGS= 2024-06-03T06:01:22.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.28700/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-06-03T06:01:22.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-03T06:01:22.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-06-03T06:01:22.784 INFO:tasks.workunit.client.0.smithi039.stdout:{"success":true}{ 2024-06-03T06:01:22.784 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "objxxx", 2024-06-03T06:01:22.784 INFO:tasks.workunit.client.0.smithi039.stdout: "clones": [ 2024-06-03T06:01:22.785 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:01:22.785 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-06-03T06:01:22.785 INFO:tasks.workunit.client.0.smithi039.stdout: "snapshots": [ 2024-06-03T06:01:22.785 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:01:22.785 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-06-03T06:01:22.785 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "snap01" 2024-06-03T06:01:22.785 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T06:01:22.785 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:01:22.785 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-06-03T06:01:22.785 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "snap02" 2024-06-03T06:01:22.785 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-06-03T06:01:22.785 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-06-03T06:01:22.785 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 6, 2024-06-03T06:01:22.785 INFO:tasks.workunit.client.0.smithi039.stdout: "overlaps": [] 2024-06-03T06:01:22.786 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T06:01:22.786 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:01:22.786 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 3, 2024-06-03T06:01:22.786 INFO:tasks.workunit.client.0.smithi039.stdout: "snapshots": [ 2024-06-03T06:01:22.786 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:01:22.786 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 3, 2024-06-03T06:01:22.786 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "snap13" 2024-06-03T06:01:22.786 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-06-03T06:01:22.786 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-06-03T06:01:22.786 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 5, 2024-06-03T06:01:22.786 INFO:tasks.workunit.client.0.smithi039.stdout: "overlaps": [] 2024-06-03T06:01:22.786 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 5, 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: "snapshots": [ 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 4, 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "snap24" 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 5, 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "snap25" 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 6, 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: "overlaps": [] 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:01:22.787 INFO:tasks.workunit.client.0.smithi039.stdout: "id": "head", 2024-06-03T06:01:22.788 INFO:tasks.workunit.client.0.smithi039.stdout: "snapshots": [], 2024-06-03T06:01:22.788 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 6 2024-06-03T06:01:22.788 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-06-03T06:01:22.788 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-06-03T06:01:22.788 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:01:22.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-06-03T06:01:22.884 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T06:01:22.884 INFO:tasks.workunit.client.0.smithi039.stdout: "deep": true, 2024-06-03T06:01:22.884 INFO:tasks.workunit.client.0.smithi039.stdout: "must": true, 2024-06-03T06:01:22.884 INFO:tasks.workunit.client.0.smithi039.stdout: "stamp": "0.000000" 2024-06-03T06:01:22.884 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:01:22.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-06-03T06:01:25.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-06-03T06:01:26.181 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:01:26.182 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:01:15.593398+0000 0'0 21:16 [1,2,0] 1 [1,2,0] 1 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:14.291230+0000 0 0 2024-06-03T06:01:26.182 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:01:15.593430+0000 0'0 24:24 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:14.291230+0000 0 0 2024-06-03T06:01:26.182 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:01:15.501002+0000 0'0 26:30 [2,0,1] 2 [2,0,1] 2 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:14.291230+0000 0 0 2024-06-03T06:01:26.182 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:01:15.593269+0000 0'0 21:16 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:14.291230+0000 0 0 2024-06-03T06:01:26.182 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:26.182 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:01:26.182 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:01:26.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-06-03T06:01:26.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-06-03T06:01:26.464 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:01:26.465 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-06-03T06:01:22.957783+0000 26'7 26:42 [1,2,0] 1 [1,2,0] 1 26'7 2024-06-03T06:01:22.957703+0000 26'7 2024-06-03T06:01:22.957703+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:22.957703+0000 4 0 2024-06-03T06:01:26.465 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:01:15.593430+0000 0'0 24:24 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:14.291230+0000 0 0 2024-06-03T06:01:26.465 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:01:15.501002+0000 0'0 26:30 [2,0,1] 2 [2,0,1] 2 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:14.291230+0000 0 0 2024-06-03T06:01:26.465 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:01:15.593269+0000 0'0 26:28 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:14.291230+0000 0 0 2024-06-03T06:01:26.465 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:26.465 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:01:26.465 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:01:26.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-06-03T06:01:26.981 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T06:01:26.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-06-03T06:01:27.517 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T06:01:27.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-06-03T06:01:32.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-06-03T06:01:32.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-06-03T06:01:32.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:01:32.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:01:32.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:01:32.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:01:32.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:01:32.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:01:32.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-03T06:01:32.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-06-03T06:01:32.869 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.868+0000 7f3a6fa3f980 1 bdev(0x55ebc2aba000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-03T06:01:32.869 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.868+0000 7f3a6fa3f980 0 bdev(0x55ebc2aba000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-06-03T06:01:32.869 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.868+0000 7f3a6fa3f980 1 bdev(0x55ebc2aba000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-06-03T06:01:32.869 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.868+0000 7f3a6fa3f980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-06-03T06:01:32.869 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.868+0000 7f3a6fa3f980 1 bdev(0x55ebc2aba700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-03T06:01:32.869 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.868+0000 7f3a6fa3f980 0 bdev(0x55ebc2aba700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-06-03T06:01:32.869 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.869+0000 7f3a6fa3f980 1 bdev(0x55ebc2aba700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-06-03T06:01:32.869 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.869+0000 7f3a6fa3f980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-06-03T06:01:32.869 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.869+0000 7f3a6fa3f980 1 bluefs mount 2024-06-03T06:01:32.875 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.875+0000 7f3a6fa3f980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-03T06:01:32.884 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.884+0000 7f3a6fa3f980 1 bluefs mount shared_bdev_used = 0 2024-06-03T06:01:32.884 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.884+0000 7f3a6fa3f980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-03T06:01:32.887 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.886+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-03T06:01:32.887 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.886+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-03T06:01:32.887 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.886+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-03T06:01:32.887 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.887+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-03T06:01:32.887 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.887+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-03T06:01:32.887 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.887+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-03T06:01:32.887 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.887+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-03T06:01:32.888 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.887+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-03T06:01:32.888 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.887+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-03T06:01:32.888 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.887+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-03T06:01:32.888 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.887+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-03T06:01:32.888 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.887+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-03T06:01:32.922 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.922+0000 7f3a6fa3f980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-03T06:01:32.922 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.922+0000 7f3a6fa3f980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-06-03T06:01:32.922 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.922+0000 7f3a6fa3f980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-06-03T06:01:32.922 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.922+0000 7f3a6fa3f980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-06-03T06:01:32.922 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.922+0000 7f3a6fa3f980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-06-03T06:01:32.922 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.922+0000 7f3a6fa3f980 1 freelist init 2024-06-03T06:01:32.922 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.922+0000 7f3a6fa3f980 1 freelist _read_cfg 2024-06-03T06:01:32.923 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.923+0000 7f3a6fa3f980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 22 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.01088e-07 2024-06-03T06:01:32.923 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.923+0000 7f3a6fa3f980 1 bluefs umount 2024-06-03T06:01:32.923 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:32.923+0000 7f3a6fa3f980 1 bdev(0x55ebc2aba700 td/osd-mapper/0/block) close 2024-06-03T06:01:33.131 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.131+0000 7f3a6fa3f980 1 bdev(0x55ebc2aba700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-03T06:01:33.132 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.131+0000 7f3a6fa3f980 0 bdev(0x55ebc2aba700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-06-03T06:01:33.132 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.131+0000 7f3a6fa3f980 1 bdev(0x55ebc2aba700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-06-03T06:01:33.132 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.131+0000 7f3a6fa3f980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-06-03T06:01:33.132 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.131+0000 7f3a6fa3f980 1 bluefs mount 2024-06-03T06:01:33.132 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.131+0000 7f3a6fa3f980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-03T06:01:33.137 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.137+0000 7f3a6fa3f980 1 bluefs mount shared_bdev_used = 27459584 2024-06-03T06:01:33.137 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.137+0000 7f3a6fa3f980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-03T06:01:33.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.139+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-03T06:01:33.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.139+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-03T06:01:33.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.139+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-03T06:01:33.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.140+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-03T06:01:33.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.140+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-03T06:01:33.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.140+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-03T06:01:33.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.140+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-03T06:01:33.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.140+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-03T06:01:33.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.140+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-03T06:01:33.141 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.140+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-03T06:01:33.141 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.140+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-03T06:01:33.141 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.140+0000 7f3a6fa3f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-03T06:01:33.844 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.844+0000 7f3a6fa3f980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-03T06:01:33.845 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-06-03T06:01:33.845 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 0b 05 58 02 00 00 56 2d 63 df 99 49 45 e5 98 3b |..X...V-c..IE..;| 2024-06-03T06:01:33.846 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 00 95 33 c6 80 1f 00 00 00 00 1c 00 00 00 00 00 |..3.............| 2024-06-03T06:01:33.846 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-06-03T06:01:33.846 INFO:tasks.workunit.client.0.smithi039.stdout:* 2024-06-03T06:01:33.846 INFO:tasks.workunit.client.0.smithi039.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-06-03T06:01:33.846 INFO:tasks.workunit.client.0.smithi039.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-06-03T06:01:33.846 INFO:tasks.workunit.client.0.smithi039.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-06-03T06:01:33.846 INFO:tasks.workunit.client.0.smithi039.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-06-03T06:01:33.846 INFO:tasks.workunit.client.0.smithi039.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-06-03T06:01:33.846 INFO:tasks.workunit.client.0.smithi039.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-06-03T06:01:33.846 INFO:tasks.workunit.client.0.smithi039.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-06-03T06:01:33.846 INFO:tasks.workunit.client.0.smithi039.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-06-03T06:01:33.846 INFO:tasks.workunit.client.0.smithi039.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-06-03T06:01:33.846 INFO:tasks.workunit.client.0.smithi039.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-06-03T06:01:33.847 INFO:tasks.workunit.client.0.smithi039.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-06-03T06:01:33.847 INFO:tasks.workunit.client.0.smithi039.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-06-03T06:01:33.847 INFO:tasks.workunit.client.0.smithi039.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-06-03T06:01:33.847 INFO:tasks.workunit.client.0.smithi039.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-06-03T06:01:33.847 INFO:tasks.workunit.client.0.smithi039.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-06-03T06:01:33.847 INFO:tasks.workunit.client.0.smithi039.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-06-03T06:01:33.847 INFO:tasks.workunit.client.0.smithi039.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-06-03T06:01:33.847 INFO:tasks.workunit.client.0.smithi039.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-06-03T06:01:33.847 INFO:tasks.workunit.client.0.smithi039.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-06-03T06:01:33.847 INFO:tasks.workunit.client.0.smithi039.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-06-03T06:01:33.847 INFO:tasks.workunit.client.0.smithi039.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-06-03T06:01:33.847 INFO:tasks.workunit.client.0.smithi039.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-06-03T06:01:33.847 INFO:tasks.workunit.client.0.smithi039.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-06-03T06:01:33.848 INFO:tasks.workunit.client.0.smithi039.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-06-03T06:01:33.848 INFO:tasks.workunit.client.0.smithi039.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-06-03T06:01:33.848 INFO:tasks.workunit.client.0.smithi039.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-06-03T06:01:33.848 INFO:tasks.workunit.client.0.smithi039.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-06-03T06:01:33.848 INFO:tasks.workunit.client.0.smithi039.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-06-03T06:01:33.848 INFO:tasks.workunit.client.0.smithi039.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-06-03T06:01:33.848 INFO:tasks.workunit.client.0.smithi039.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-06-03T06:01:33.848 INFO:tasks.workunit.client.0.smithi039.stdout:00000220 72 65 1c 00 00 00 05 00 00 00 34 4b 9b 28 d7 4e |re........4K.(.N| 2024-06-03T06:01:33.848 INFO:tasks.workunit.client.0.smithi039.stdout:00000230 49 37 b9 35 27 9b 9c cb 0e 9d 00 00 00 00 00 00 |I7.5'...........| 2024-06-03T06:01:33.848 INFO:tasks.workunit.client.0.smithi039.stdout:00000240 00 00 1c 00 00 00 e5 5b 5d 66 00 48 a6 10 01 00 |.......[]f.H....| 2024-06-03T06:01:33.848 INFO:tasks.workunit.client.0.smithi039.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-06-03T06:01:33.848 INFO:tasks.workunit.client.0.smithi039.stdout:0000025e 2024-06-03T06:01:33.848 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:33.849 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-06-03T06:01:33.849 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:33.849 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:33.849 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-03T06:01:33.849 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-06-03T06:01:33.849 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-03T06:01:33.849 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-06-03T06:01:33.849 INFO:tasks.workunit.client.0.smithi039.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-03T06:01:33.849 INFO:tasks.workunit.client.0.smithi039.stdout:00000047 2024-06-03T06:01:33.849 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:33.849 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:33.849 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-06-03T06:01:33.850 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-06-03T06:01:33.850 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-03T06:01:33.850 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-06-03T06:01:33.850 INFO:tasks.workunit.client.0.smithi039.stdout:0000003f 2024-06-03T06:01:33.850 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:33.850 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:33.850 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-03T06:01:33.850 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-06-03T06:01:33.850 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-03T06:01:33.850 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-06-03T06:01:33.850 INFO:tasks.workunit.client.0.smithi039.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-03T06:01:33.850 INFO:tasks.workunit.client.0.smithi039.stdout:00000047 2024-06-03T06:01:33.850 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:33.851 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:33.851 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:33.851 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:33.851 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:33.851 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:33.851 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:33.851 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:33.851 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:33.851 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:33.851 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:33.851 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:33.851 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:33.851 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:33.852 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:33.852 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:33.852 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:33.852 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:33.852 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:33.852 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:33.852 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:33.852 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:33.852 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:33.852 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:33.852 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:33.852 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:33.852 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:33.853 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:33.853 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:33.853 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:33.853 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:33.853 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:33.853 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:33.853 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:33.853 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:33.853 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:33.853 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-06-03T06:01:33.853 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:33.853 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-06-03T06:01:33.853 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:33.854 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.845+0000 7f3a6fa3f980 1 bluefs umount 2024-06-03T06:01:33.854 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:33.845+0000 7f3a6fa3f980 1 bdev(0x55ebc2aba700 td/osd-mapper/0/block) close 2024-06-03T06:01:34.092 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.093+0000 7f3a6fa3f980 1 freelist shutdown 2024-06-03T06:01:34.093 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.093+0000 7f3a6fa3f980 1 bdev(0x55ebc2aba000 td/osd-mapper/0/block) close 2024-06-03T06:01:34.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-03T06:01:34.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-06-03T06:01:34.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-06-03T06:01:34.161 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.160+0000 7f6fc4041980 1 bdev(0x56119b52a000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-03T06:01:34.161 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.160+0000 7f6fc4041980 0 bdev(0x56119b52a000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-06-03T06:01:34.161 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.161+0000 7f6fc4041980 1 bdev(0x56119b52a000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-06-03T06:01:34.161 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.161+0000 7f6fc4041980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-06-03T06:01:34.161 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.161+0000 7f6fc4041980 1 bdev(0x56119b52a700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-03T06:01:34.161 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.161+0000 7f6fc4041980 0 bdev(0x56119b52a700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-06-03T06:01:34.162 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.161+0000 7f6fc4041980 1 bdev(0x56119b52a700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-06-03T06:01:34.162 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.161+0000 7f6fc4041980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-06-03T06:01:34.162 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.161+0000 7f6fc4041980 1 bluefs mount 2024-06-03T06:01:34.173 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.173+0000 7f6fc4041980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-03T06:01:34.181 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.181+0000 7f6fc4041980 1 bluefs mount shared_bdev_used = 0 2024-06-03T06:01:34.181 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.181+0000 7f6fc4041980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-03T06:01:34.185 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.184+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-03T06:01:34.185 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.184+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-03T06:01:34.185 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.184+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-03T06:01:34.185 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.184+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-03T06:01:34.185 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.184+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-03T06:01:34.185 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.184+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-03T06:01:34.185 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.184+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-03T06:01:34.185 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.184+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-03T06:01:34.185 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.184+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-03T06:01:34.185 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.184+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-03T06:01:34.185 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.184+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-03T06:01:34.185 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.184+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-03T06:01:34.220 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.219+0000 7f6fc4041980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-03T06:01:34.220 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.220+0000 7f6fc4041980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-06-03T06:01:34.220 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.220+0000 7f6fc4041980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-06-03T06:01:34.220 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.220+0000 7f6fc4041980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-06-03T06:01:34.220 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.220+0000 7f6fc4041980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-06-03T06:01:34.220 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.220+0000 7f6fc4041980 1 freelist init 2024-06-03T06:01:34.220 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.220+0000 7f6fc4041980 1 freelist _read_cfg 2024-06-03T06:01:34.221 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.221+0000 7f6fc4041980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 13 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 4.57765e-07 2024-06-03T06:01:34.221 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.221+0000 7f6fc4041980 1 bluefs umount 2024-06-03T06:01:34.221 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.221+0000 7f6fc4041980 1 bdev(0x56119b52a700 td/osd-mapper/2/block) close 2024-06-03T06:01:34.431 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.431+0000 7f6fc4041980 1 bdev(0x56119b52a700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-03T06:01:34.431 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.431+0000 7f6fc4041980 0 bdev(0x56119b52a700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-06-03T06:01:34.431 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.431+0000 7f6fc4041980 1 bdev(0x56119b52a700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-06-03T06:01:34.431 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.431+0000 7f6fc4041980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-06-03T06:01:34.431 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.431+0000 7f6fc4041980 1 bluefs mount 2024-06-03T06:01:34.447 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.447+0000 7f6fc4041980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-03T06:01:34.458 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.458+0000 7f6fc4041980 1 bluefs mount shared_bdev_used = 27459584 2024-06-03T06:01:34.458 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.458+0000 7f6fc4041980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-03T06:01:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.460+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-03T06:01:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.460+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-03T06:01:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.460+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-03T06:01:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.460+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-03T06:01:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.460+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-03T06:01:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.460+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-03T06:01:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.460+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-03T06:01:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.461+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-03T06:01:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.461+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-03T06:01:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.461+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-03T06:01:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.461+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-03T06:01:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:34.461+0000 7f6fc4041980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-03T06:01:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:35.307+0000 7f6fc4041980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-03T06:01:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:35.308+0000 7f6fc4041980 1 bluefs umount 2024-06-03T06:01:35.308 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:35.308+0000 7f6fc4041980 1 bdev(0x56119b52a700 td/osd-mapper/2/block) close 2024-06-03T06:01:35.544 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:35.544+0000 7f6fc4041980 1 freelist shutdown 2024-06-03T06:01:35.544 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:35.544+0000 7f6fc4041980 1 bdev(0x56119b52a000 td/osd-mapper/2/block) close 2024-06-03T06:01:35.677 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:35.678 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:35.678 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:35.678 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:35.678 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:35.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-03T06:01:35.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-06-03T06:01:35.679 INFO:tasks.workunit.client.0.smithi039.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-06-03T06:01:35.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-03T06:01:35.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-06-03T06:01:36.251 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-06-03T06:01:36.251 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 0b 05 58 02 00 00 56 2d 63 df 99 49 45 e5 98 3b |..X...V-c..IE..;| 2024-06-03T06:01:36.251 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 00 95 33 c6 80 1f 02 00 00 00 1c 00 00 00 00 00 |..3.............| 2024-06-03T06:01:36.251 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-06-03T06:01:36.251 INFO:tasks.workunit.client.0.smithi039.stdout:* 2024-06-03T06:01:36.251 INFO:tasks.workunit.client.0.smithi039.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-06-03T06:01:36.251 INFO:tasks.workunit.client.0.smithi039.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-06-03T06:01:36.251 INFO:tasks.workunit.client.0.smithi039.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-06-03T06:01:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-06-03T06:01:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-06-03T06:01:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-06-03T06:01:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-06-03T06:01:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-06-03T06:01:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-06-03T06:01:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-06-03T06:01:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-06-03T06:01:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-06-03T06:01:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-06-03T06:01:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-06-03T06:01:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-06-03T06:01:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-06-03T06:01:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-06-03T06:01:36.253 INFO:tasks.workunit.client.0.smithi039.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-06-03T06:01:36.253 INFO:tasks.workunit.client.0.smithi039.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-06-03T06:01:36.253 INFO:tasks.workunit.client.0.smithi039.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-06-03T06:01:36.253 INFO:tasks.workunit.client.0.smithi039.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-06-03T06:01:36.253 INFO:tasks.workunit.client.0.smithi039.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-06-03T06:01:36.253 INFO:tasks.workunit.client.0.smithi039.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-06-03T06:01:36.253 INFO:tasks.workunit.client.0.smithi039.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-06-03T06:01:36.253 INFO:tasks.workunit.client.0.smithi039.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-06-03T06:01:36.253 INFO:tasks.workunit.client.0.smithi039.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-06-03T06:01:36.253 INFO:tasks.workunit.client.0.smithi039.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-06-03T06:01:36.253 INFO:tasks.workunit.client.0.smithi039.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-06-03T06:01:36.253 INFO:tasks.workunit.client.0.smithi039.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-06-03T06:01:36.253 INFO:tasks.workunit.client.0.smithi039.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-06-03T06:01:36.254 INFO:tasks.workunit.client.0.smithi039.stdout:00000220 72 65 1c 00 00 00 0f 00 00 00 9f 46 a9 5d e2 69 |re.........F.].i| 2024-06-03T06:01:36.254 INFO:tasks.workunit.client.0.smithi039.stdout:00000230 42 31 8c be 9e 24 04 18 bf 27 00 00 00 00 00 00 |B1...$...'......| 2024-06-03T06:01:36.254 INFO:tasks.workunit.client.0.smithi039.stdout:00000240 00 00 1c 00 00 00 28 5c 5d 66 5b 2e ed 27 01 00 |......(\]f[..'..| 2024-06-03T06:01:36.254 INFO:tasks.workunit.client.0.smithi039.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-06-03T06:01:36.254 INFO:tasks.workunit.client.0.smithi039.stdout:0000025e 2024-06-03T06:01:36.254 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:36.254 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-06-03T06:01:36.254 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:36.254 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:36.254 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-03T06:01:36.254 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-06-03T06:01:36.254 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-03T06:01:36.255 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-06-03T06:01:36.255 INFO:tasks.workunit.client.0.smithi039.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-03T06:01:36.255 INFO:tasks.workunit.client.0.smithi039.stdout:00000047 2024-06-03T06:01:36.255 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:36.255 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:36.255 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-06-03T06:01:36.255 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-06-03T06:01:36.255 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-03T06:01:36.255 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-06-03T06:01:36.255 INFO:tasks.workunit.client.0.smithi039.stdout:0000003f 2024-06-03T06:01:36.255 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:36.255 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:36.255 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-03T06:01:36.256 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-06-03T06:01:36.256 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-03T06:01:36.256 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-06-03T06:01:36.256 INFO:tasks.workunit.client.0.smithi039.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-03T06:01:36.256 INFO:tasks.workunit.client.0.smithi039.stdout:00000047 2024-06-03T06:01:36.256 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:36.256 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:36.256 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:36.256 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:36.256 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:36.256 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:36.256 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:36.256 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:36.257 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:36.257 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:36.257 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:36.257 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:36.257 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:36.257 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:36.257 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:36.257 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:36.257 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:36.257 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:36.257 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:36.257 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:36.257 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:36.258 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:36.258 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:36.258 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:36.258 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:36.258 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:36.258 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:36.258 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:36.258 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:36.258 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:36.258 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:36.258 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:36.258 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:36.258 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:36.259 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:36.259 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:36.259 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-06-03T06:01:36.259 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:36.259 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-06-03T06:01:36.259 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:36.730 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-06-03T06:01:36.730 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-06-03T06:01:36.730 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-06-03T06:01:36.730 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2024-06-03T06:01:38.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2024-06-03T06:01:38.274 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2024-06-03T06:01:38.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2024-06-03T06:01:38.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-03T06:01:38.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2024-06-03T06:01:38.276 INFO:tasks.workunit.client.0.smithi039.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2024-06-03T06:01:38.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2024-06-03T06:01:38.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-06-03T06:01:38.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-06-03T06:01:38.302 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.301+0000 7f15d0a37980 1 bdev(0x5639a699c000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-03T06:01:38.302 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.301+0000 7f15d0a37980 0 bdev(0x5639a699c000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-06-03T06:01:38.302 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.301+0000 7f15d0a37980 1 bdev(0x5639a699c000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-06-03T06:01:38.302 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.301+0000 7f15d0a37980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-06-03T06:01:38.303 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.301+0000 7f15d0a37980 1 bdev(0x5639a699c700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-03T06:01:38.303 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.301+0000 7f15d0a37980 0 bdev(0x5639a699c700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-06-03T06:01:38.303 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.301+0000 7f15d0a37980 1 bdev(0x5639a699c700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-06-03T06:01:38.303 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.301+0000 7f15d0a37980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-06-03T06:01:38.303 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.301+0000 7f15d0a37980 1 bluefs mount 2024-06-03T06:01:38.308 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.307+0000 7f15d0a37980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-03T06:01:38.317 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.316+0000 7f15d0a37980 1 bluefs mount shared_bdev_used = 0 2024-06-03T06:01:38.317 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.316+0000 7f15d0a37980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-03T06:01:38.321 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.319+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-03T06:01:38.321 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.319+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-03T06:01:38.321 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.319+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-03T06:01:38.321 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.319+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-03T06:01:38.321 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.319+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-03T06:01:38.321 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.319+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-03T06:01:38.321 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.319+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-03T06:01:38.321 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.319+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-03T06:01:38.321 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.319+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-03T06:01:38.321 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.319+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-03T06:01:38.321 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.319+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-03T06:01:38.321 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.319+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-03T06:01:38.327 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.325+0000 7f15d0a37980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-03T06:01:38.327 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.325+0000 7f15d0a37980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-06-03T06:01:38.327 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.325+0000 7f15d0a37980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-06-03T06:01:38.327 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.325+0000 7f15d0a37980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-06-03T06:01:38.327 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.325+0000 7f15d0a37980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-06-03T06:01:38.328 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.326+0000 7f15d0a37980 1 freelist init 2024-06-03T06:01:38.328 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.326+0000 7f15d0a37980 1 freelist _read_cfg 2024-06-03T06:01:38.328 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.326+0000 7f15d0a37980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 22 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.01088e-07 2024-06-03T06:01:38.328 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.326+0000 7f15d0a37980 1 bluefs umount 2024-06-03T06:01:38.328 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.326+0000 7f15d0a37980 1 bdev(0x5639a699c700 td/osd-mapper/0/block) close 2024-06-03T06:01:38.564 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.563+0000 7f15d0a37980 1 bdev(0x5639a699c700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-03T06:01:38.564 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.563+0000 7f15d0a37980 0 bdev(0x5639a699c700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-06-03T06:01:38.564 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.563+0000 7f15d0a37980 1 bdev(0x5639a699c700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-06-03T06:01:38.564 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.563+0000 7f15d0a37980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-06-03T06:01:38.564 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.563+0000 7f15d0a37980 1 bluefs mount 2024-06-03T06:01:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.573+0000 7f15d0a37980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-03T06:01:38.583 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.582+0000 7f15d0a37980 1 bluefs mount shared_bdev_used = 9568256 2024-06-03T06:01:38.583 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.582+0000 7f15d0a37980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-03T06:01:38.586 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.584+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-03T06:01:38.586 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.584+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-03T06:01:38.586 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.584+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-03T06:01:38.586 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.584+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-03T06:01:38.586 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.584+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-03T06:01:38.586 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.584+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-03T06:01:38.586 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.584+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-03T06:01:38.586 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.584+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-03T06:01:38.586 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.584+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-03T06:01:38.586 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.585+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-03T06:01:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.585+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-03T06:01:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.585+0000 7f15d0a37980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-03T06:01:38.940 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.940+0000 7f15d0a37980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-03T06:01:38.941 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-06-03T06:01:38.942 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 0b 05 58 02 00 00 56 2d 63 df 99 49 45 e5 98 3b |..X...V-c..IE..;| 2024-06-03T06:01:38.942 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 00 95 33 c6 80 1f 00 00 00 00 1c 00 00 00 00 00 |..3.............| 2024-06-03T06:01:38.942 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-06-03T06:01:38.942 INFO:tasks.workunit.client.0.smithi039.stdout:* 2024-06-03T06:01:38.942 INFO:tasks.workunit.client.0.smithi039.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-06-03T06:01:38.942 INFO:tasks.workunit.client.0.smithi039.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-06-03T06:01:38.942 INFO:tasks.workunit.client.0.smithi039.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-06-03T06:01:38.942 INFO:tasks.workunit.client.0.smithi039.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-06-03T06:01:38.942 INFO:tasks.workunit.client.0.smithi039.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-06-03T06:01:38.942 INFO:tasks.workunit.client.0.smithi039.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-06-03T06:01:38.942 INFO:tasks.workunit.client.0.smithi039.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-06-03T06:01:38.942 INFO:tasks.workunit.client.0.smithi039.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-06-03T06:01:38.942 INFO:tasks.workunit.client.0.smithi039.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-06-03T06:01:38.943 INFO:tasks.workunit.client.0.smithi039.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-06-03T06:01:38.943 INFO:tasks.workunit.client.0.smithi039.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-06-03T06:01:38.943 INFO:tasks.workunit.client.0.smithi039.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-06-03T06:01:38.943 INFO:tasks.workunit.client.0.smithi039.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-06-03T06:01:38.943 INFO:tasks.workunit.client.0.smithi039.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-06-03T06:01:38.943 INFO:tasks.workunit.client.0.smithi039.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-06-03T06:01:38.943 INFO:tasks.workunit.client.0.smithi039.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-06-03T06:01:38.943 INFO:tasks.workunit.client.0.smithi039.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-06-03T06:01:38.943 INFO:tasks.workunit.client.0.smithi039.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-06-03T06:01:38.943 INFO:tasks.workunit.client.0.smithi039.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-06-03T06:01:38.943 INFO:tasks.workunit.client.0.smithi039.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-06-03T06:01:38.943 INFO:tasks.workunit.client.0.smithi039.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-06-03T06:01:38.944 INFO:tasks.workunit.client.0.smithi039.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-06-03T06:01:38.944 INFO:tasks.workunit.client.0.smithi039.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-06-03T06:01:38.944 INFO:tasks.workunit.client.0.smithi039.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-06-03T06:01:38.944 INFO:tasks.workunit.client.0.smithi039.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-06-03T06:01:38.944 INFO:tasks.workunit.client.0.smithi039.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-06-03T06:01:38.944 INFO:tasks.workunit.client.0.smithi039.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-06-03T06:01:38.944 INFO:tasks.workunit.client.0.smithi039.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-06-03T06:01:38.944 INFO:tasks.workunit.client.0.smithi039.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-06-03T06:01:38.944 INFO:tasks.workunit.client.0.smithi039.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-06-03T06:01:38.944 INFO:tasks.workunit.client.0.smithi039.stdout:00000220 72 65 1c 00 00 00 05 00 00 00 34 4b 9b 28 d7 4e |re........4K.(.N| 2024-06-03T06:01:38.944 INFO:tasks.workunit.client.0.smithi039.stdout:00000230 49 37 b9 35 27 9b 9c cb 0e 9d 00 00 00 00 00 00 |I7.5'...........| 2024-06-03T06:01:38.944 INFO:tasks.workunit.client.0.smithi039.stdout:00000240 00 00 1c 00 00 00 e5 5b 5d 66 00 48 a6 10 01 00 |.......[]f.H....| 2024-06-03T06:01:38.944 INFO:tasks.workunit.client.0.smithi039.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-06-03T06:01:38.945 INFO:tasks.workunit.client.0.smithi039.stdout:0000025e 2024-06-03T06:01:38.945 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:38.945 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-06-03T06:01:38.945 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:38.945 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:38.945 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-03T06:01:38.945 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-06-03T06:01:38.945 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-03T06:01:38.945 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-06-03T06:01:38.945 INFO:tasks.workunit.client.0.smithi039.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-03T06:01:38.945 INFO:tasks.workunit.client.0.smithi039.stdout:00000047 2024-06-03T06:01:38.945 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:38.946 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:38.946 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-06-03T06:01:38.946 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-06-03T06:01:38.946 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-03T06:01:38.946 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-06-03T06:01:38.946 INFO:tasks.workunit.client.0.smithi039.stdout:0000003f 2024-06-03T06:01:38.946 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:38.946 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:38.946 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-03T06:01:38.946 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-06-03T06:01:38.946 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-03T06:01:38.946 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-06-03T06:01:38.947 INFO:tasks.workunit.client.0.smithi039.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-03T06:01:38.947 INFO:tasks.workunit.client.0.smithi039.stdout:00000047 2024-06-03T06:01:38.947 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:38.947 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:38.947 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:38.947 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:38.947 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:38.947 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:38.947 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:38.947 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:38.947 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:38.947 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:38.947 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:38.948 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:38.948 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:38.948 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:38.948 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:38.948 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:38.948 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:38.948 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:38.948 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:38.948 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:38.948 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:38.948 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:38.948 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:38.948 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:38.948 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:38.949 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:38.949 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:38.949 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:38.949 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:38.949 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:38.949 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:38.949 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:38.949 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:38.949 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:38.949 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:38.949 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:38.949 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-06-03T06:01:38.949 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:38.949 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-06-03T06:01:38.950 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:38.950 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.940+0000 7f15d0a37980 1 bluefs umount 2024-06-03T06:01:38.950 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:38.940+0000 7f15d0a37980 1 bdev(0x5639a699c700 td/osd-mapper/0/block) close 2024-06-03T06:01:39.178 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:39.177+0000 7f15d0a37980 1 freelist shutdown 2024-06-03T06:01:39.178 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:39.177+0000 7f15d0a37980 1 bdev(0x5639a699c000 td/osd-mapper/0/block) close 2024-06-03T06:01:39.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-06-03T06:01:39.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-06-03T06:01:39.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-06-03T06:01:40.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:40.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-03T06:01:40.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-06-03T06:01:40.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-06-03T06:01:40.117 INFO:tasks.workunit.client.0.smithi039.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:40.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-06-03T06:01:40.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.i9VJLrTFmY_the_val 2024-06-03T06:01:40.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-03T06:01:40.119 INFO:tasks.workunit.client.0.smithi039.stdout:Value dumped in: /tmp/tmp.i9VJLrTFmY_the_val 2024-06-03T06:01:40.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.i9VJLrTFmY_the_val' 2024-06-03T06:01:40.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.i9VJLrTFmY_the_val 2024-06-03T06:01:40.713 INFO:tasks.workunit.client.0.smithi039.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-06-03T06:01:41.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-03T06:01:41.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.i9VJLrTFmY_the_val 2024-06-03T06:01:41.175 INFO:tasks.workunit.client.0.smithi039.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-06-03T06:01:41.175 INFO:tasks.workunit.client.0.smithi039.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-06-03T06:01:41.175 INFO:tasks.workunit.client.0.smithi039.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-06-03T06:01:41.176 INFO:tasks.workunit.client.0.smithi039.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-06-03T06:01:41.176 INFO:tasks.workunit.client.0.smithi039.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-06-03T06:01:41.176 INFO:tasks.workunit.client.0.smithi039.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-06-03T06:01:41.176 INFO:tasks.workunit.client.0.smithi039.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-06-03T06:01:41.176 INFO:tasks.workunit.client.0.smithi039.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-06-03T06:01:41.176 INFO:tasks.workunit.client.0.smithi039.stdout:0000073 2024-06-03T06:01:41.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-06-03T06:01:41.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:41.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.i9VJLrTFmY_the_val 2024-06-03T06:01:43.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-03T06:01:43.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-03T06:01:43.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-06-03T06:01:43.013 INFO:tasks.workunit.client.0.smithi039.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-06-03T06:01:43.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-06-03T06:01:43.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-06-03T06:01:43.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-06-03T06:01:43.039 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.037+0000 7f7eadc3b980 1 bdev(0x5641cb6fe000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-06-03T06:01:43.039 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.037+0000 7f7eadc3b980 0 bdev(0x5641cb6fe000 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-06-03T06:01:43.039 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.038+0000 7f7eadc3b980 1 bdev(0x5641cb6fe000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-06-03T06:01:43.039 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.038+0000 7f7eadc3b980 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-06-03T06:01:43.039 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.038+0000 7f7eadc3b980 1 bdev(0x5641cb6fe700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-06-03T06:01:43.039 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.038+0000 7f7eadc3b980 0 bdev(0x5641cb6fe700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-06-03T06:01:43.039 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.038+0000 7f7eadc3b980 1 bdev(0x5641cb6fe700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-06-03T06:01:43.039 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.038+0000 7f7eadc3b980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-06-03T06:01:43.040 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.038+0000 7f7eadc3b980 1 bluefs mount 2024-06-03T06:01:43.047 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.047+0000 7f7eadc3b980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-03T06:01:43.057 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.056+0000 7f7eadc3b980 1 bluefs mount shared_bdev_used = 0 2024-06-03T06:01:43.057 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.056+0000 7f7eadc3b980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-03T06:01:43.060 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.058+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-03T06:01:43.060 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.058+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-03T06:01:43.060 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.058+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-03T06:01:43.060 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.058+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-03T06:01:43.060 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.058+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-03T06:01:43.060 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.059+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-03T06:01:43.061 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.059+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-03T06:01:43.061 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.059+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-03T06:01:43.061 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.059+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-03T06:01:43.061 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.059+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-03T06:01:43.061 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.059+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-03T06:01:43.061 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.059+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-03T06:01:43.066 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.065+0000 7f7eadc3b980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-03T06:01:43.066 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.065+0000 7f7eadc3b980 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-06-03T06:01:43.066 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.065+0000 7f7eadc3b980 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-06-03T06:01:43.066 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.065+0000 7f7eadc3b980 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-06-03T06:01:43.067 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.065+0000 7f7eadc3b980 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-06-03T06:01:43.067 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.065+0000 7f7eadc3b980 1 freelist init 2024-06-03T06:01:43.067 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.065+0000 7f7eadc3b980 1 freelist _read_cfg 2024-06-03T06:01:43.067 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.066+0000 7f7eadc3b980 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 16 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 5.72206e-07 2024-06-03T06:01:43.067 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.066+0000 7f7eadc3b980 1 bluefs umount 2024-06-03T06:01:43.068 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.066+0000 7f7eadc3b980 1 bdev(0x5641cb6fe700 td/osd-mapper/1/block) close 2024-06-03T06:01:43.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.301+0000 7f7eadc3b980 1 bdev(0x5641cb6fe700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-06-03T06:01:43.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.301+0000 7f7eadc3b980 0 bdev(0x5641cb6fe700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-06-03T06:01:43.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.301+0000 7f7eadc3b980 1 bdev(0x5641cb6fe700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-06-03T06:01:43.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.301+0000 7f7eadc3b980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-06-03T06:01:43.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.301+0000 7f7eadc3b980 1 bluefs mount 2024-06-03T06:01:43.306 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.305+0000 7f7eadc3b980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-03T06:01:43.314 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.313+0000 7f7eadc3b980 1 bluefs mount shared_bdev_used = 9568256 2024-06-03T06:01:43.314 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.313+0000 7f7eadc3b980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-03T06:01:43.317 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.315+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-03T06:01:43.317 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.315+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-03T06:01:43.317 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.315+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-03T06:01:43.317 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.316+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-03T06:01:43.317 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.316+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-03T06:01:43.317 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.316+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-03T06:01:43.317 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.316+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-03T06:01:43.318 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.316+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-03T06:01:43.318 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.316+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-03T06:01:43.318 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.316+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-03T06:01:43.318 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.316+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-03T06:01:43.318 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.316+0000 7f7eadc3b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-03T06:01:43.569 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.568+0000 7f7eadc3b980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-03T06:01:43.571 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-06-03T06:01:43.571 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 0b 05 58 02 00 00 56 2d 63 df 99 49 45 e5 98 3b |..X...V-c..IE..;| 2024-06-03T06:01:43.571 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 00 95 33 c6 80 1f 01 00 00 00 1c 00 00 00 00 00 |..3.............| 2024-06-03T06:01:43.571 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-06-03T06:01:43.572 INFO:tasks.workunit.client.0.smithi039.stdout:* 2024-06-03T06:01:43.572 INFO:tasks.workunit.client.0.smithi039.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-06-03T06:01:43.572 INFO:tasks.workunit.client.0.smithi039.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-06-03T06:01:43.572 INFO:tasks.workunit.client.0.smithi039.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-06-03T06:01:43.572 INFO:tasks.workunit.client.0.smithi039.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-06-03T06:01:43.572 INFO:tasks.workunit.client.0.smithi039.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-06-03T06:01:43.572 INFO:tasks.workunit.client.0.smithi039.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-06-03T06:01:43.572 INFO:tasks.workunit.client.0.smithi039.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-06-03T06:01:43.572 INFO:tasks.workunit.client.0.smithi039.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-06-03T06:01:43.572 INFO:tasks.workunit.client.0.smithi039.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-06-03T06:01:43.572 INFO:tasks.workunit.client.0.smithi039.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-06-03T06:01:43.572 INFO:tasks.workunit.client.0.smithi039.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-06-03T06:01:43.572 INFO:tasks.workunit.client.0.smithi039.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-06-03T06:01:43.572 INFO:tasks.workunit.client.0.smithi039.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-06-03T06:01:43.573 INFO:tasks.workunit.client.0.smithi039.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-06-03T06:01:43.573 INFO:tasks.workunit.client.0.smithi039.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-06-03T06:01:43.573 INFO:tasks.workunit.client.0.smithi039.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-06-03T06:01:43.573 INFO:tasks.workunit.client.0.smithi039.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-06-03T06:01:43.573 INFO:tasks.workunit.client.0.smithi039.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-06-03T06:01:43.573 INFO:tasks.workunit.client.0.smithi039.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-06-03T06:01:43.573 INFO:tasks.workunit.client.0.smithi039.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-06-03T06:01:43.573 INFO:tasks.workunit.client.0.smithi039.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-06-03T06:01:43.573 INFO:tasks.workunit.client.0.smithi039.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-06-03T06:01:43.573 INFO:tasks.workunit.client.0.smithi039.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-06-03T06:01:43.573 INFO:tasks.workunit.client.0.smithi039.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-06-03T06:01:43.573 INFO:tasks.workunit.client.0.smithi039.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-06-03T06:01:43.573 INFO:tasks.workunit.client.0.smithi039.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-06-03T06:01:43.573 INFO:tasks.workunit.client.0.smithi039.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-06-03T06:01:43.574 INFO:tasks.workunit.client.0.smithi039.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-06-03T06:01:43.574 INFO:tasks.workunit.client.0.smithi039.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-06-03T06:01:43.574 INFO:tasks.workunit.client.0.smithi039.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-06-03T06:01:43.574 INFO:tasks.workunit.client.0.smithi039.stdout:00000220 72 65 1c 00 00 00 0a 00 00 00 e3 b3 7c c4 be 5d |re..........|..]| 2024-06-03T06:01:43.574 INFO:tasks.workunit.client.0.smithi039.stdout:00000230 42 69 b7 74 fb 63 8e 2c 89 8a 00 00 00 00 00 00 |Bi.t.c.,........| 2024-06-03T06:01:43.574 INFO:tasks.workunit.client.0.smithi039.stdout:00000240 00 00 1c 00 00 00 06 5c 5d 66 ae f6 12 36 01 00 |.......\]f...6..| 2024-06-03T06:01:43.574 INFO:tasks.workunit.client.0.smithi039.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-06-03T06:01:43.574 INFO:tasks.workunit.client.0.smithi039.stdout:0000025e 2024-06-03T06:01:43.574 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:43.574 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-06-03T06:01:43.574 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:43.574 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:43.574 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-03T06:01:43.575 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-06-03T06:01:43.575 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-03T06:01:43.575 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-06-03T06:01:43.575 INFO:tasks.workunit.client.0.smithi039.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-03T06:01:43.575 INFO:tasks.workunit.client.0.smithi039.stdout:00000047 2024-06-03T06:01:43.575 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:43.575 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:43.575 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-06-03T06:01:43.575 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-06-03T06:01:43.575 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-03T06:01:43.575 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-06-03T06:01:43.575 INFO:tasks.workunit.client.0.smithi039.stdout:0000003f 2024-06-03T06:01:43.575 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:43.575 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:43.576 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-03T06:01:43.576 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-06-03T06:01:43.576 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-03T06:01:43.576 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-06-03T06:01:43.576 INFO:tasks.workunit.client.0.smithi039.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-03T06:01:43.577 INFO:tasks.workunit.client.0.smithi039.stdout:00000047 2024-06-03T06:01:43.577 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:43.577 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:43.577 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:43.577 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:43.577 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:43.577 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:43.577 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:43.577 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:43.577 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:43.577 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:43.578 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:43.578 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:43.578 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:43.578 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:43.578 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:43.578 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:43.578 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:43.578 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:43.578 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:43.578 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:43.578 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:43.578 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:43.578 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:43.579 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:43.579 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:43.579 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:43.579 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:43.579 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:43.579 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:43.579 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:43.579 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:43.579 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:43.579 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:43.579 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:43.579 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:43.579 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:43.579 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-06-03T06:01:43.579 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:43.580 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-06-03T06:01:43.580 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:43.580 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.569+0000 7f7eadc3b980 1 bluefs umount 2024-06-03T06:01:43.580 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.569+0000 7f7eadc3b980 1 bdev(0x5641cb6fe700 td/osd-mapper/1/block) close 2024-06-03T06:01:43.811 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.811+0000 7f7eadc3b980 1 freelist shutdown 2024-06-03T06:01:43.811 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:43.811+0000 7f7eadc3b980 1 bdev(0x5641cb6fe000 td/osd-mapper/1/block) close 2024-06-03T06:01:44.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-06-03T06:01:44.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-06-03T06:01:44.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-06-03T06:01:45.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:45.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-03T06:01:45.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-06-03T06:01:45.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-06-03T06:01:45.106 INFO:tasks.workunit.client.0.smithi039.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:45.107 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-06-03T06:01:45.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.gIVTcSptyO_the_val 2024-06-03T06:01:45.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-03T06:01:45.108 INFO:tasks.workunit.client.0.smithi039.stdout:Value dumped in: /tmp/tmp.gIVTcSptyO_the_val 2024-06-03T06:01:45.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.gIVTcSptyO_the_val' 2024-06-03T06:01:45.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.gIVTcSptyO_the_val 2024-06-03T06:01:45.672 INFO:tasks.workunit.client.0.smithi039.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-06-03T06:01:46.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-03T06:01:46.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.gIVTcSptyO_the_val 2024-06-03T06:01:46.154 INFO:tasks.workunit.client.0.smithi039.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-06-03T06:01:46.155 INFO:tasks.workunit.client.0.smithi039.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-06-03T06:01:46.155 INFO:tasks.workunit.client.0.smithi039.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-06-03T06:01:46.155 INFO:tasks.workunit.client.0.smithi039.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-06-03T06:01:46.155 INFO:tasks.workunit.client.0.smithi039.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-06-03T06:01:46.155 INFO:tasks.workunit.client.0.smithi039.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-06-03T06:01:46.155 INFO:tasks.workunit.client.0.smithi039.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-06-03T06:01:46.155 INFO:tasks.workunit.client.0.smithi039.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-06-03T06:01:46.155 INFO:tasks.workunit.client.0.smithi039.stdout:0000073 2024-06-03T06:01:46.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-06-03T06:01:46.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:47.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.gIVTcSptyO_the_val 2024-06-03T06:01:48.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-03T06:01:48.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-03T06:01:48.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-06-03T06:01:48.242 INFO:tasks.workunit.client.0.smithi039.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-06-03T06:01:48.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-06-03T06:01:48.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-06-03T06:01:48.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-06-03T06:01:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.267+0000 7ff83b566980 1 bdev(0x5629af20e000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-03T06:01:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.268+0000 7ff83b566980 0 bdev(0x5629af20e000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-06-03T06:01:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.268+0000 7ff83b566980 1 bdev(0x5629af20e000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-06-03T06:01:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.268+0000 7ff83b566980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-06-03T06:01:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.268+0000 7ff83b566980 1 bdev(0x5629af20e700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-03T06:01:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.268+0000 7ff83b566980 0 bdev(0x5629af20e700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-06-03T06:01:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.268+0000 7ff83b566980 1 bdev(0x5629af20e700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-06-03T06:01:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.268+0000 7ff83b566980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-06-03T06:01:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.268+0000 7ff83b566980 1 bluefs mount 2024-06-03T06:01:48.279 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.280+0000 7ff83b566980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-03T06:01:48.297 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.297+0000 7ff83b566980 1 bluefs mount shared_bdev_used = 0 2024-06-03T06:01:48.297 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.297+0000 7ff83b566980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-03T06:01:48.300 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.299+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-03T06:01:48.300 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.299+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-03T06:01:48.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.299+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-03T06:01:48.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.300+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-03T06:01:48.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.300+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-03T06:01:48.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.300+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-03T06:01:48.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.300+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-03T06:01:48.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.300+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-03T06:01:48.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.300+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-03T06:01:48.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.300+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-03T06:01:48.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.300+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-03T06:01:48.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.300+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-03T06:01:48.306 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.306+0000 7ff83b566980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-03T06:01:48.307 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.306+0000 7ff83b566980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-06-03T06:01:48.307 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.306+0000 7ff83b566980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-06-03T06:01:48.307 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.306+0000 7ff83b566980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-06-03T06:01:48.307 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.306+0000 7ff83b566980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-06-03T06:01:48.307 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.307+0000 7ff83b566980 1 freelist init 2024-06-03T06:01:48.307 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.307+0000 7ff83b566980 1 freelist _read_cfg 2024-06-03T06:01:48.308 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.307+0000 7ff83b566980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 13 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 4.57765e-07 2024-06-03T06:01:48.308 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.307+0000 7ff83b566980 1 bluefs umount 2024-06-03T06:01:48.308 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.307+0000 7ff83b566980 1 bdev(0x5629af20e700 td/osd-mapper/2/block) close 2024-06-03T06:01:48.530 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.530+0000 7ff83b566980 1 bdev(0x5629af20e700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-03T06:01:48.530 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.530+0000 7ff83b566980 0 bdev(0x5629af20e700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-06-03T06:01:48.530 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.530+0000 7ff83b566980 1 bdev(0x5629af20e700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-06-03T06:01:48.530 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.530+0000 7ff83b566980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-06-03T06:01:48.530 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.530+0000 7ff83b566980 1 bluefs mount 2024-06-03T06:01:48.537 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.538+0000 7ff83b566980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-03T06:01:48.546 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.546+0000 7ff83b566980 1 bluefs mount shared_bdev_used = 9568256 2024-06-03T06:01:48.546 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.546+0000 7ff83b566980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-03T06:01:48.549 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.549+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-03T06:01:48.549 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.549+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-03T06:01:48.549 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.549+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-03T06:01:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.549+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-03T06:01:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.549+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-03T06:01:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.549+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-03T06:01:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.549+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-03T06:01:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.549+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-03T06:01:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.549+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-03T06:01:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.549+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-03T06:01:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.549+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-03T06:01:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.549+0000 7ff83b566980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-03T06:01:48.852 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.852+0000 7ff83b566980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-03T06:01:48.854 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-06-03T06:01:48.854 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 0b 05 58 02 00 00 56 2d 63 df 99 49 45 e5 98 3b |..X...V-c..IE..;| 2024-06-03T06:01:48.854 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 00 95 33 c6 80 1f 02 00 00 00 1c 00 00 00 00 00 |..3.............| 2024-06-03T06:01:48.854 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-06-03T06:01:48.854 INFO:tasks.workunit.client.0.smithi039.stdout:* 2024-06-03T06:01:48.854 INFO:tasks.workunit.client.0.smithi039.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-06-03T06:01:48.854 INFO:tasks.workunit.client.0.smithi039.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-06-03T06:01:48.854 INFO:tasks.workunit.client.0.smithi039.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-06-03T06:01:48.854 INFO:tasks.workunit.client.0.smithi039.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-06-03T06:01:48.855 INFO:tasks.workunit.client.0.smithi039.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-06-03T06:01:48.855 INFO:tasks.workunit.client.0.smithi039.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-06-03T06:01:48.855 INFO:tasks.workunit.client.0.smithi039.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-06-03T06:01:48.855 INFO:tasks.workunit.client.0.smithi039.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-06-03T06:01:48.855 INFO:tasks.workunit.client.0.smithi039.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-06-03T06:01:48.855 INFO:tasks.workunit.client.0.smithi039.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-06-03T06:01:48.855 INFO:tasks.workunit.client.0.smithi039.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-06-03T06:01:48.855 INFO:tasks.workunit.client.0.smithi039.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-06-03T06:01:48.855 INFO:tasks.workunit.client.0.smithi039.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-06-03T06:01:48.855 INFO:tasks.workunit.client.0.smithi039.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-06-03T06:01:48.855 INFO:tasks.workunit.client.0.smithi039.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-06-03T06:01:48.855 INFO:tasks.workunit.client.0.smithi039.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-06-03T06:01:48.855 INFO:tasks.workunit.client.0.smithi039.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-06-03T06:01:48.856 INFO:tasks.workunit.client.0.smithi039.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-06-03T06:01:48.856 INFO:tasks.workunit.client.0.smithi039.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-06-03T06:01:48.856 INFO:tasks.workunit.client.0.smithi039.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-06-03T06:01:48.856 INFO:tasks.workunit.client.0.smithi039.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-06-03T06:01:48.856 INFO:tasks.workunit.client.0.smithi039.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-06-03T06:01:48.856 INFO:tasks.workunit.client.0.smithi039.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-06-03T06:01:48.856 INFO:tasks.workunit.client.0.smithi039.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-06-03T06:01:48.856 INFO:tasks.workunit.client.0.smithi039.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-06-03T06:01:48.856 INFO:tasks.workunit.client.0.smithi039.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-06-03T06:01:48.856 INFO:tasks.workunit.client.0.smithi039.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-06-03T06:01:48.856 INFO:tasks.workunit.client.0.smithi039.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-06-03T06:01:48.856 INFO:tasks.workunit.client.0.smithi039.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-06-03T06:01:48.856 INFO:tasks.workunit.client.0.smithi039.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-06-03T06:01:48.857 INFO:tasks.workunit.client.0.smithi039.stdout:00000220 72 65 1c 00 00 00 0f 00 00 00 9f 46 a9 5d e2 69 |re.........F.].i| 2024-06-03T06:01:48.857 INFO:tasks.workunit.client.0.smithi039.stdout:00000230 42 31 8c be 9e 24 04 18 bf 27 00 00 00 00 00 00 |B1...$...'......| 2024-06-03T06:01:48.857 INFO:tasks.workunit.client.0.smithi039.stdout:00000240 00 00 1c 00 00 00 28 5c 5d 66 5b 2e ed 27 01 00 |......(\]f[..'..| 2024-06-03T06:01:48.857 INFO:tasks.workunit.client.0.smithi039.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-06-03T06:01:48.857 INFO:tasks.workunit.client.0.smithi039.stdout:0000025e 2024-06-03T06:01:48.857 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:48.857 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-06-03T06:01:48.857 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:48.857 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:48.857 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-03T06:01:48.857 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-06-03T06:01:48.857 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-03T06:01:48.857 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-06-03T06:01:48.857 INFO:tasks.workunit.client.0.smithi039.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-03T06:01:48.858 INFO:tasks.workunit.client.0.smithi039.stdout:00000047 2024-06-03T06:01:48.858 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:48.858 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:48.858 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-06-03T06:01:48.858 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-06-03T06:01:48.858 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-03T06:01:48.858 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-06-03T06:01:48.858 INFO:tasks.workunit.client.0.smithi039.stdout:0000003f 2024-06-03T06:01:48.858 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:48.858 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:48.858 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-03T06:01:48.858 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-06-03T06:01:48.858 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-03T06:01:48.859 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-06-03T06:01:48.859 INFO:tasks.workunit.client.0.smithi039.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-03T06:01:48.859 INFO:tasks.workunit.client.0.smithi039.stdout:00000047 2024-06-03T06:01:48.859 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:48.859 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:48.859 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:48.859 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:48.859 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:48.859 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:48.859 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:48.859 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:48.859 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:01:48.859 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:48.859 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:48.860 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:48.860 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:48.860 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:48.860 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:48.860 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:48.860 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:48.860 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:48.860 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:48.860 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:48.860 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:48.860 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:48.860 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:48.860 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:48.860 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:48.861 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:48.861 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:48.861 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:48.861 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:48.861 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:01:48.861 INFO:tasks.workunit.client.0.smithi039.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-03T06:01:48.861 INFO:tasks.workunit.client.0.smithi039.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-03T06:01:48.861 INFO:tasks.workunit.client.0.smithi039.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-03T06:01:48.861 INFO:tasks.workunit.client.0.smithi039.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-03T06:01:48.861 INFO:tasks.workunit.client.0.smithi039.stdout:0000003b 2024-06-03T06:01:48.861 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:48.861 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-06-03T06:01:48.861 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:48.861 INFO:tasks.workunit.client.0.smithi039.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-06-03T06:01:48.862 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:01:48.862 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.853+0000 7ff83b566980 1 bluefs umount 2024-06-03T06:01:48.862 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:48.853+0000 7ff83b566980 1 bdev(0x5629af20e700 td/osd-mapper/2/block) close 2024-06-03T06:01:49.073 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:49.073+0000 7ff83b566980 1 freelist shutdown 2024-06-03T06:01:49.073 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:49.073+0000 7ff83b566980 1 bdev(0x5629af20e000 td/osd-mapper/2/block) close 2024-06-03T06:01:49.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-06-03T06:01:49.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-06-03T06:01:49.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-06-03T06:01:50.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:50.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-03T06:01:50.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-06-03T06:01:50.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-06-03T06:01:50.083 INFO:tasks.workunit.client.0.smithi039.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:50.084 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-06-03T06:01:50.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.tR2NNop52G_the_val 2024-06-03T06:01:50.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-03T06:01:50.085 INFO:tasks.workunit.client.0.smithi039.stdout:Value dumped in: /tmp/tmp.tR2NNop52G_the_val 2024-06-03T06:01:50.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.tR2NNop52G_the_val' 2024-06-03T06:01:50.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.tR2NNop52G_the_val 2024-06-03T06:01:50.662 INFO:tasks.workunit.client.0.smithi039.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-06-03T06:01:51.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-03T06:01:51.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.tR2NNop52G_the_val 2024-06-03T06:01:51.135 INFO:tasks.workunit.client.0.smithi039.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-06-03T06:01:51.135 INFO:tasks.workunit.client.0.smithi039.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-06-03T06:01:51.135 INFO:tasks.workunit.client.0.smithi039.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-06-03T06:01:51.135 INFO:tasks.workunit.client.0.smithi039.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-06-03T06:01:51.135 INFO:tasks.workunit.client.0.smithi039.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-06-03T06:01:51.136 INFO:tasks.workunit.client.0.smithi039.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-06-03T06:01:51.136 INFO:tasks.workunit.client.0.smithi039.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-06-03T06:01:51.136 INFO:tasks.workunit.client.0.smithi039.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-06-03T06:01:51.136 INFO:tasks.workunit.client.0.smithi039.stdout:0000073 2024-06-03T06:01:51.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-06-03T06:01:51.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:01:52.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.tR2NNop52G_the_val 2024-06-03T06:01:53.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-03T06:01:53.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 ' 2024-06-03T06:01:53.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 2024-06-03T06:01:53.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-03T06:01:53.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-03T06:01:53.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/' 2024-06-03T06:01:53.477 INFO:tasks.workunit.client.0.smithi039.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ 2024-06-03T06:01:53.477 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-06-03T06:01:53.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-06-03T06:01:53.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-03T06:01:53.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=562d63df-9949-45e5-983b-009533c6801f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-03T06:01:53.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-06-03T06:01:53.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-06-03T06:01:53.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:01:53.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:01:53.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:01:53.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2024-06-03T06:01:53.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=562d63df-9949-45e5-983b-009533c6801f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-03T06:01:53.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:01:53.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:01:53.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:01:53.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-06-03T06:01:53.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-06-03T06:01:53.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:01:53.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:01:53.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-06-03T06:01:53.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:01:53.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:01:53.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:01:53.483 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:01:53.483 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:01:53.483 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28700 2024-06-03T06:01:53.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28700/$cluster-$name.asok' 2024-06-03T06:01:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28700/$cluster-$name.asok' 2024-06-03T06:01:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:01:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-06-03T06:01:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-06-03T06:01:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:01:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:01:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:01:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:01:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:01:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:01:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2024-06-03T06:01:53.485 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:01:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:01:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=562d63df-9949-45e5-983b-009533c6801f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28700/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:01:53.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2024-06-03T06:01:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:01:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:01:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:01:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:01:53.527 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:53.527+0000 7f7f8976e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:01:53.529 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:53.530+0000 7f7f8976e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:01:53.531 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:53.531+0000 7f7f8976e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:01:53.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:01:53.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:01:53.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:01:53.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:01:53.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:01:53.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:01:53.797 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:01:53.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:01:53.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:01:53.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:01:54.048 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:54.049+0000 7f7f8976e640 -1 Falling back to public interface 2024-06-03T06:01:54.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:01:55.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:01:55.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:01:55.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:01:55.138 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:01:55.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:01:55.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:01:55.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:01:56.379 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:56.380+0000 7f7f8976e640 -1 osd.0 28 log_to_monitors true 2024-06-03T06:01:56.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:01:56.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:01:56.448 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:01:56.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:01:56.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:01:56.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:01:56.829 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 31 up_thru 17 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/725682537,v1:127.0.0.1:6803/725682537] [v2:127.0.0.1:6804/725682537,v1:127.0.0.1:6805/725682537] exists,up 344b9b28-d74e-4937-b935-279b9ccb0e9d 2024-06-03T06:01:56.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:01:56.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:01:56.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:01:56.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-03T06:01:56.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=562d63df-9949-45e5-983b-009533c6801f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-03T06:01:56.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-06-03T06:01:56.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-06-03T06:01:56.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:01:56.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:01:56.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:01:56.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2024-06-03T06:01:56.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=562d63df-9949-45e5-983b-009533c6801f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-03T06:01:56.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:01:56.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:01:56.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:01:56.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-06-03T06:01:56.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-06-03T06:01:56.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:01:56.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:01:56.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-06-03T06:01:56.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:01:56.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:01:56.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:01:56.947 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:01:56.947 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:01:56.947 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28700 2024-06-03T06:01:56.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28700/$cluster-$name.asok' 2024-06-03T06:01:56.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28700/$cluster-$name.asok' 2024-06-03T06:01:56.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:01:56.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-06-03T06:01:56.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-06-03T06:01:56.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:01:56.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:01:56.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:01:56.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:01:56.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:01:56.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:01:56.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2024-06-03T06:01:56.948 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:01:56.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:01:56.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=562d63df-9949-45e5-983b-009533c6801f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28700/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:01:56.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2024-06-03T06:01:56.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:01:56.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:01:56.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:01:56.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:01:56.951 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:56.869+0000 7f351682c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:01:56.951 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:56.893+0000 7f351682c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:01:56.951 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:56.894+0000 7f351682c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:01:57.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:01:57.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:01:57.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:01:57.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:01:57.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:01:57.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:01:57.156 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:01:57.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:01:57.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:01:57.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:01:57.412 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:57.412+0000 7f351682c640 -1 Falling back to public interface 2024-06-03T06:01:57.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:01:58.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:01:58.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:01:58.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:01:58.499 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:01:58.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:01:58.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:01:58.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:01:59.686 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:01:59.686+0000 7f351682c640 -1 osd.1 28 log_to_monitors true 2024-06-03T06:01:59.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:01:59.813 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:01:59.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:01:59.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:01:59.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:01:59.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:02:00.179 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 34 up_thru 17 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/4017308038,v1:127.0.0.1:6811/4017308038] [v2:127.0.0.1:6812/4017308038,v1:127.0.0.1:6813/4017308038] exists,up e3b37cc4-be5d-4269-b774-fb638e2c898a 2024-06-03T06:02:00.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:02:00.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:02:00.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:02:00.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-03T06:02:00.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=562d63df-9949-45e5-983b-009533c6801f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-03T06:02:00.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-06-03T06:02:00.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-06-03T06:02:00.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:02:00.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-03T06:02:00.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:02:00.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2024-06-03T06:02:00.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=562d63df-9949-45e5-983b-009533c6801f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-03T06:02:00.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:02:00.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:02:00.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:02:00.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-06-03T06:02:00.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-06-03T06:02:00.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:02:00.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:02:00.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-06-03T06:02:00.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:02:00.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:02:00.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:02:00.184 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:02:00.184 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:02:00.184 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28700 2024-06-03T06:02:00.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28700/$cluster-$name.asok' 2024-06-03T06:02:00.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28700/$cluster-$name.asok' 2024-06-03T06:02:00.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:02:00.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-06-03T06:02:00.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-06-03T06:02:00.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:02:00.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:02:00.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:02:00.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:02:00.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:02:00.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:02:00.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2024-06-03T06:02:00.185 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:02:00.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-03T06:02:00.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=562d63df-9949-45e5-983b-009533c6801f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28700/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:02:00.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2024-06-03T06:02:00.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-03T06:02:00.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:02:00.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:02:00.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:02:00.219 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:00.220+0000 7fdbe9166640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:00.222 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:00.222+0000 7fdbe9166640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:00.223 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:00.223+0000 7fdbe9166640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:00.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-03T06:02:00.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:02:00.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:02:00.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:02:00.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:02:00.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:02:00.590 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:02:00.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:02:00.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:02:00.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:02:00.749 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:00.749+0000 7fdbe9166640 -1 Falling back to public interface 2024-06-03T06:02:00.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:02:01.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:02:01.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:02:01.967 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:02:01.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:02:01.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:02:01.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:02:02.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:02:03.118 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:03.118+0000 7fdbe9166640 -1 osd.2 28 log_to_monitors true 2024-06-03T06:02:03.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:02:03.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:02:03.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:02:03.279 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:02:03.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:02:03.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:02:03.772 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 38 up_thru 17 down_at 29 last_clean_interval [15,28) [v2:127.0.0.1:6818/1184976895,v1:127.0.0.1:6819/1184976895] [v2:127.0.0.1:6820/1184976895,v1:127.0.0.1:6821/1184976895] exists,up 9f46a95d-e269-4231-8cbe-9e240418bf27 2024-06-03T06:02:03.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:02:03.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:02:03.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:02:03.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-06-03T06:02:04.775 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-06-03T06:02:04.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-06-03T06:02:04.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-03T06:02:04.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-06-03T06:02:04.892 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T06:02:04.892 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "19.0.0-2946-g3d89accf", 2024-06-03T06:02:04.892 INFO:tasks.workunit.client.0.smithi039.stdout: "release": "squid", 2024-06-03T06:02:04.893 INFO:tasks.workunit.client.0.smithi039.stdout: "release_type": "rc" 2024-06-03T06:02:04.893 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:02:04.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-03T06:02:04.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-06-03T06:02:05.042 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T06:02:05.042 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "19.0.0-2946-g3d89accf", 2024-06-03T06:02:05.042 INFO:tasks.workunit.client.0.smithi039.stdout: "release": "squid", 2024-06-03T06:02:05.042 INFO:tasks.workunit.client.0.smithi039.stdout: "release_type": "rc" 2024-06-03T06:02:05.042 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:02:05.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-03T06:02:05.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-06-03T06:02:05.152 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T06:02:05.152 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "19.0.0-2946-g3d89accf", 2024-06-03T06:02:05.152 INFO:tasks.workunit.client.0.smithi039.stdout: "release": "squid", 2024-06-03T06:02:05.152 INFO:tasks.workunit.client.0.smithi039.stdout: "release_type": "rc" 2024-06-03T06:02:05.152 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:02:05.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-06-03T06:02:05.190 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T06:02:05.190 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "objxxx", 2024-06-03T06:02:05.190 INFO:tasks.workunit.client.0.smithi039.stdout: "clones": [ 2024-06-03T06:02:05.190 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:02:05.191 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-06-03T06:02:05.191 INFO:tasks.workunit.client.0.smithi039.stdout: "snapshots": [ 2024-06-03T06:02:05.191 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:02:05.191 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-06-03T06:02:05.191 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "snap01" 2024-06-03T06:02:05.191 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T06:02:05.191 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:02:05.191 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-06-03T06:02:05.191 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "snap02" 2024-06-03T06:02:05.191 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-06-03T06:02:05.191 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-06-03T06:02:05.191 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 6, 2024-06-03T06:02:05.191 INFO:tasks.workunit.client.0.smithi039.stdout: "overlaps": [] 2024-06-03T06:02:05.192 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T06:02:05.192 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:02:05.192 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 3, 2024-06-03T06:02:05.192 INFO:tasks.workunit.client.0.smithi039.stdout: "snapshots": [ 2024-06-03T06:02:05.192 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:02:05.192 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 3, 2024-06-03T06:02:05.192 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "snap13" 2024-06-03T06:02:05.192 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-06-03T06:02:05.192 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-06-03T06:02:05.192 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 5, 2024-06-03T06:02:05.192 INFO:tasks.workunit.client.0.smithi039.stdout: "overlaps": [] 2024-06-03T06:02:05.192 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T06:02:05.192 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:02:05.193 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 5, 2024-06-03T06:02:05.193 INFO:tasks.workunit.client.0.smithi039.stdout: "snapshots": [ 2024-06-03T06:02:05.193 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:02:05.193 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 4, 2024-06-03T06:02:05.193 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "snap24" 2024-06-03T06:02:05.193 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T06:02:05.193 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:02:05.193 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 5, 2024-06-03T06:02:05.193 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "snap25" 2024-06-03T06:02:05.193 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-06-03T06:02:05.193 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-06-03T06:02:05.194 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 6, 2024-06-03T06:02:05.194 INFO:tasks.workunit.client.0.smithi039.stdout: "overlaps": [] 2024-06-03T06:02:05.194 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T06:02:05.194 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-06-03T06:02:05.194 INFO:tasks.workunit.client.0.smithi039.stdout: "id": "head", 2024-06-03T06:02:05.194 INFO:tasks.workunit.client.0.smithi039.stdout: "snapshots": [], 2024-06-03T06:02:05.194 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 6 2024-06-03T06:02:05.194 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-06-03T06:02:05.194 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-06-03T06:02:05.194 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:02:05.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-06-03T06:02:05.577 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is unset 2024-06-03T06:02:05.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-06-03T06:02:06.138 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is unset 2024-06-03T06:02:06.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-06-03T06:02:06.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-06-03T06:02:06.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-06-03T06:02:06.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-06-03T06:02:06.782 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:02:06.783 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-06-03T06:02:04.645929+0000 26'7 40:63 [1,2,0] 1 [1,2,0] 1 26'7 2024-06-03T06:01:22.957703+0000 26'7 2024-06-03T06:01:22.957703+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:22.957703+0000 4 0 2024-06-03T06:02:06.783 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-06-03T06:02:01.131555+0000 0'0 36:25 [0,1] 0 [0,1] 0 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:02:06.783 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-06-03T06:02:01.131521+0000 0'0 36:17 [0,1] 0 [0,1] 0 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:14.291230+0000 0 0 2024-06-03T06:02:06.783 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:02:04.645924+0000 0'0 40:33 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:14.291230+0000 0 0 2024-06-03T06:02:06.783 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:02:06.783 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:02:06.783 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:02:06.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-06-03T06:02:08.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-06-03T06:02:08.904 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T06:02:08.905 INFO:tasks.workunit.client.0.smithi039.stdout: "deep": true, 2024-06-03T06:02:08.905 INFO:tasks.workunit.client.0.smithi039.stdout: "must": true, 2024-06-03T06:02:08.905 INFO:tasks.workunit.client.0.smithi039.stdout: "stamp": "0.000000" 2024-06-03T06:02:08.905 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:02:08.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-06-03T06:02:13.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-06-03T06:02:14.200 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:02:14.201 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-06-03T06:02:09.746203+0000 26'7 42:77 [1,2,0] 1 [1,2,0] 1 26'7 2024-06-03T06:02:09.746109+0000 26'7 2024-06-03T06:02:09.746109+0000 0 1 periodic scrub scheduled @ 2024-06-04T06:02:09.746109+0000 4 0 2024-06-03T06:02:14.201 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:02:04.206851+0000 0'0 42:42 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:14.291230+0000 0 0 2024-06-03T06:02:14.201 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:02:04.247255+0000 0'0 41:28 [2,0,1] 2 [2,0,1] 2 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:14.291230+0000 0 0 2024-06-03T06:02:14.201 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:02:04.645924+0000 0'0 42:37 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:14.291230+0000 0 0 2024-06-03T06:02:14.201 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:02:14.201 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:02:14.202 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:02:14.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-03T06:02:14.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-06-03T06:02:14.213 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:02:09.745+0000 7f34f85f5640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-06-03T06:02:14.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-06-03T06:02:14.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-06-03T06:02:14.217 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-06-03T06:02:14.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-06-03T06:02:14.218 INFO:tasks.workunit.client.0.smithi039.stdout:prev count: 1 2024-06-03T06:02:14.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-06-03T06:02:14.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-06-03T06:02:14.316 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T06:02:14.316 INFO:tasks.workunit.client.0.smithi039.stdout: "deep": true, 2024-06-03T06:02:14.316 INFO:tasks.workunit.client.0.smithi039.stdout: "must": true, 2024-06-03T06:02:14.316 INFO:tasks.workunit.client.0.smithi039.stdout: "stamp": "0.000000" 2024-06-03T06:02:14.316 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:02:14.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-06-03T06:02:19.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-06-03T06:02:19.608 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:02:19.608 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-06-03T06:02:14.591601+0000 26'7 42:86 [1,2,0] 1 [1,2,0] 1 26'7 2024-06-03T06:02:14.591522+0000 26'7 2024-06-03T06:02:14.591522+0000 0 1 periodic scrub scheduled @ 2024-06-04T06:02:14.591522+0000 4 0 2024-06-03T06:02:19.608 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:02:04.206851+0000 0'0 42:42 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:14.291230+0000 0 0 2024-06-03T06:02:19.608 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:02:04.247255+0000 0'0 42:30 [2,0,1] 2 [2,0,1] 2 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:14.291230+0000 0 0 2024-06-03T06:02:19.609 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:02:04.645924+0000 0'0 42:37 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-03T06:01:14.291230+0000 0'0 2024-06-03T06:01:14.291230+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:01:14.291230+0000 0 0 2024-06-03T06:02:19.609 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:02:19.609 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:02:19.609 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:02:19.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-03T06:02:19.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-06-03T06:02:19.620 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:02:09.745+0000 7f34f85f5640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-06-03T06:02:19.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-06-03T06:02:19.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-06-03T06:02:19.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-06-03T06:02:19.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-03T06:02:19.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-06-03T06:02:19.623 INFO:tasks.workunit.client.0.smithi039.stdout:current count: 1 2024-06-03T06:02:19.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-06-03T06:02:19.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-06-03T06:02:19.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:02:19.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:02:19.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:02:19.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:02:19.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:02:19.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:02:19.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-06-03T06:02:19.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-03T06:02:19.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-06-03T06:02:19.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-06-03T06:02:19.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-06-03T06:02:21.469 INFO:tasks.workunit.client.0.smithi039.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:02:21.469 INFO:tasks.workunit.client.0.smithi039.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-03T06:02:21.469 INFO:tasks.workunit.client.0.smithi039.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-06-03T06:02:21.469 INFO:tasks.workunit.client.0.smithi039.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-03T06:02:21.469 INFO:tasks.workunit.client.0.smithi039.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:02:21.469 INFO:tasks.workunit.client.0.smithi039.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-03T06:02:21.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-06-03T06:02:21.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-06-03T06:02:21.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-06-03T06:02:21.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-06-03T06:02:22.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-06-03T06:02:22.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-06-03T06:02:22.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-06-03T06:02:22.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-06-03T06:02:22.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-06-03T06:02:22.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:02:22.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-06-03T06:02:22.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:02:22.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:02:22.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:02:22.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:02:22.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:02:22.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:02:22.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:02:22.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:02:22.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:02:22.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:02:22.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:02:22.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:02:22.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:02:22.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:02:22.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:02:22.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:02:22.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:02:22.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:02:22.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-06-03T06:02:22.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:02:22.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:02:22.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28700 2024-06-03T06:02:22.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28700 2024-06-03T06:02:22.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:02:22.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:02:22.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-06-03T06:02:22.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-mapper 0 2024-06-03T06:02:22.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-06-03T06:02:22.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-03T06:02:22.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-06-03T06:02:22.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:02:22.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:02:22.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:02:22.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:02:22.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:02:22.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:02:22.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:02:22.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:02:22.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:02:22.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:02:22.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:02:22.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:02:22.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:02:22.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:02:22.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:02:22.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:02:22.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:02:22.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-03T06:02:22.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-06-03T06:02:22.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:02:22.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:02:22.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28700 2024-06-03T06:02:22.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28700 2024-06-03T06:02:22.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:02:22.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:02:22.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-06-03T06:02:22.412 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-03T06:02:22.413 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-03T06:02:22.483 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-06-03T06:02:22.483 DEBUG:teuthology.orchestra.run.smithi039:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-06-03T06:02:22.549 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-03T06:02:22.554 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-03T06:02:22.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-06-03T06:02:22.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-06-03T06:02:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-06-03T06:02:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-06-03T06:02:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-06-03T06:02:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-06-03T06:02:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-06-03T06:02:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-06-03T06:02:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-recovery-scrub 2024-06-03T06:02:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-06-03T06:02:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-06-03T06:02:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-06-03T06:02:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-06-03T06:02:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-06-03T06:02:22.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-06-03T06:02:22.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none ' 2024-06-03T06:02:22.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-06-03T06:02:22.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-06-03T06:02:22.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-06-03T06:02:22.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-06-03T06:02:22.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-03T06:02:22.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-06-03T06:02:22.559 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_recovery_scrub_2' 2024-06-03T06:02:22.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-06-03T06:02:22.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-06-03T06:02:22.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-06-03T06:02:22.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-06-03T06:02:22.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.44967 2024-06-03T06:02:22.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-06-03T06:02:22.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-06-03T06:02:22.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-06-03T06:02:22.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-06-03T06:02:22.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-06-03T06:02:22.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-06-03T06:02:22.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-06-03T06:02:22.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-06-03T06:02:22.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:02:22.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-06-03T06:02:22.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:02:22.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:02:22.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:02:22.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:02:22.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:02:22.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:02:22.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:02:22.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:02:22.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:02:22.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:02:22.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:02:22.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:02:22.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:02:22.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:02:22.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:02:22.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:02:22.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:02:22.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:02:22.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-06-03T06:02:22.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:02:22.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:02:22.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:02:22.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44967 2024-06-03T06:02:22.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:02:22.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:02:22.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-06-03T06:02:22.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:02:22.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:02:22.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:02:22.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.44967 2024-06-03T06:02:22.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:02:22.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-03T06:02:22.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-03T06:02:22.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:02:22.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-06-03T06:02:22.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T06:02:22.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-06-03T06:02:22.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T06:02:22.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T06:02:22.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T06:02:22.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-06-03T06:02:22.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T06:02:22.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T06:02:22.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:02:22.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:02:22.957 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:02:22.957 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:02:22.957 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:02:22.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:02:22.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T06:02:23.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T06:02:23.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T06:02:23.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:02:23.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:02:23.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T06:02:23.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T06:02:23.177 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:02:23.177 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:02:23.177 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:02:23.177 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:02:23.177 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:02:23.177 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:02:23.177 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44967/ceph-mon.a.asok 2024-06-03T06:02:23.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:02:23.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44967/ceph-mon.a.asok config get fsid 2024-06-03T06:02:23.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T06:02:23.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:02:23.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:02:23.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T06:02:23.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T06:02:23.268 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:02:23.268 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:02:23.268 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:02:23.268 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:02:23.268 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:02:23.268 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:02:23.268 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44967/ceph-mon.a.asok 2024-06-03T06:02:23.269 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:02:23.269 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44967/ceph-mon.a.asok config get mon_host 2024-06-03T06:02:23.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-06-03T06:02:23.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-06-03T06:02:23.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T06:02:23.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T06:02:23.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T06:02:23.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-06-03T06:02:23.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T06:02:24.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T06:02:24.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:02:24.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:02:24.168 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:02:24.168 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:02:24.169 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:02:24.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:02:24.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:02:24.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:02:24.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-06-03T06:02:24.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-06-03T06:02:24.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-06-03T06:02:24.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-06-03T06:02:24.200 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-06-03T06:02:24.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-06-03T06:02:24.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:02:24.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-06-03T06:02:24.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-03T06:02:24.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:02:24.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T06:02:24.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:02:24.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-06-03T06:02:24.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-03T06:02:24.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:02:24.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:02:24.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:02:24.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-06-03T06:02:24.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-06-03T06:02:24.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:02:24.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:02:24.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-03T06:02:24.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:02:24.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:02:24.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:02:24.207 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:02:24.207 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:02:24.207 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:02:24.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:02:24.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:02:24.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:02:24.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:02:24.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:02:24.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-03T06:02:24.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-03T06:02:24.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:02:24.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:02:24.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:02:24.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:02:24.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:02:24.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-06-03T06:02:24.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-06-03T06:02:24.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:02:24.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ff492604-0dc5-4c50-b317-48ce10faeb07 2024-06-03T06:02:24.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ff492604-0dc5-4c50-b317-48ce10faeb07' 2024-06-03T06:02:24.208 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 ff492604-0dc5-4c50-b317-48ce10faeb07 2024-06-03T06:02:24.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:02:24.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBwXF1mNclfDRAA00n9UKQ87P0G4DdzGwziuA== 2024-06-03T06:02:24.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBwXF1mNclfDRAA00n9UKQ87P0G4DdzGwziuA=="}' 2024-06-03T06:02:24.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ff492604-0dc5-4c50-b317-48ce10faeb07 -i td/osd-recovery-scrub/0/new.json 2024-06-03T06:02:24.499 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:02:24.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-06-03T06:02:24.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBwXF1mNclfDRAA00n9UKQ87P0G4DdzGwziuA== --osd-uuid ff492604-0dc5-4c50-b317-48ce10faeb07 2024-06-03T06:02:24.540 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:24.540+0000 7fdfcd5b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:24.550 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:24.550+0000 7fdfcd5b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:24.551 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:24.551+0000 7fdfcd5b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:24.551 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:24.551+0000 7fdfcd5b4640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-03T06:02:24.551 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:24.551+0000 7fdfcd5b4640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-03T06:02:24.551 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:24.551+0000 7fdfcd5b4640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-03T06:02:24.625 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:24.625+0000 7fdfcd5b4640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-06-03T06:02:30.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-06-03T06:02:30.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:02:30.143 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:02:30.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T06:02:30.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:02:30.520 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:02:30.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T06:02:30.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-06-03T06:02:30.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:02:30.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:02:30.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:02:30.548 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:30.547+0000 7f8e60fa3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:30.560 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:30.559+0000 7f8e60fa3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:30.561 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:30.560+0000 7f8e60fa3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:30.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T06:02:30.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:02:30.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:02:30.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:02:30.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:02:30.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:02:30.811 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:02:30.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:02:30.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:02:30.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:02:31.082 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:31.081+0000 7f8e60fa3640 -1 Falling back to public interface 2024-06-03T06:02:31.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:02:32.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:02:32.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:02:32.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:02:32.101 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:02:32.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:02:32.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:02:32.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:02:33.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:02:33.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:02:33.407 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:02:33.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:02:33.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:02:33.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:02:33.426 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:33.426+0000 7f8e60fa3640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:02:33.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:02:34.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:02:34.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:02:34.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:02:34.733 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:02:34.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:02:34.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:02:35.030 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2233089654,v1:127.0.0.1:6803/2233089654] [v2:127.0.0.1:6804/2233089654,v1:127.0.0.1:6805/2233089654] exists,up ff492604-0dc5-4c50-b317-48ce10faeb07 2024-06-03T06:02:35.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:02:35.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:02:35.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:02:35.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:02:35.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-06-03T06:02:35.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-03T06:02:35.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:02:35.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T06:02:35.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:02:35.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-06-03T06:02:35.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-03T06:02:35.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:02:35.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:02:35.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:02:35.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-06-03T06:02:35.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-06-03T06:02:35.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:02:35.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:02:35.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-03T06:02:35.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:02:35.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:02:35.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:02:35.036 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:02:35.036 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:02:35.036 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:02:35.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:02:35.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:02:35.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:02:35.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:02:35.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:02:35.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-03T06:02:35.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-03T06:02:35.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:02:35.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:02:35.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:02:35.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:02:35.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:02:35.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-06-03T06:02:35.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-06-03T06:02:35.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:02:35.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d6e5ead-3347-4d7f-aa44-7074ab052365 2024-06-03T06:02:35.039 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 2d6e5ead-3347-4d7f-aa44-7074ab052365 2024-06-03T06:02:35.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2d6e5ead-3347-4d7f-aa44-7074ab052365' 2024-06-03T06:02:35.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:02:35.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB7XF1msQAaAxAAO3dC/qlqCzw+kQVyaRbk4Q== 2024-06-03T06:02:35.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB7XF1msQAaAxAAO3dC/qlqCzw+kQVyaRbk4Q=="}' 2024-06-03T06:02:35.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d6e5ead-3347-4d7f-aa44-7074ab052365 -i td/osd-recovery-scrub/1/new.json 2024-06-03T06:02:35.421 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:02:35.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-06-03T06:02:35.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQB7XF1msQAaAxAAO3dC/qlqCzw+kQVyaRbk4Q== --osd-uuid 2d6e5ead-3347-4d7f-aa44-7074ab052365 2024-06-03T06:02:35.463 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:35.462+0000 7f7c03a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:35.497 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:35.496+0000 7f7c03a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:35.498 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:35.497+0000 7f7c03a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:35.498 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:35.498+0000 7f7c03a0c640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-03T06:02:35.499 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:35.498+0000 7f7c03a0c640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-03T06:02:35.499 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:35.498+0000 7f7c03a0c640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-03T06:02:35.572 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:35.572+0000 7f7c03a0c640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-06-03T06:02:41.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-06-03T06:02:41.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:02:41.315 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:02:41.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T06:02:41.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:02:41.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T06:02:41.691 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:02:41.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-06-03T06:02:41.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:02:41.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:02:41.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:02:41.719 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:41.719+0000 7fa28db01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:41.721 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:41.721+0000 7fa28db01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:41.722 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:41.722+0000 7fa28db01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:41.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T06:02:41.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:02:41.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:02:41.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:02:41.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:02:41.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:02:41.979 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:02:41.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:02:41.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:02:41.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:02:42.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:02:42.267 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:42.268+0000 7fa28db01640 -1 Falling back to public interface 2024-06-03T06:02:43.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:02:43.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:02:43.266 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:02:43.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:02:43.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:02:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:02:43.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:02:44.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:02:44.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:02:44.571 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:02:44.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:02:44.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:02:44.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:02:44.653 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:44.654+0000 7fa28db01640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:02:44.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:02:45.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:02:45.873 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:02:45.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:02:45.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:02:45.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:02:45.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:02:46.194 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/9581506,v1:127.0.0.1:6811/9581506] [v2:127.0.0.1:6812/9581506,v1:127.0.0.1:6813/9581506] exists,up 2d6e5ead-3347-4d7f-aa44-7074ab052365 2024-06-03T06:02:46.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:02:46.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:02:46.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:02:46.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:02:46.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-06-03T06:02:46.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-03T06:02:46.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:02:46.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T06:02:46.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:02:46.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-06-03T06:02:46.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-03T06:02:46.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:02:46.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:02:46.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:02:46.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-06-03T06:02:46.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-06-03T06:02:46.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:02:46.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:02:46.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-03T06:02:46.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:02:46.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:02:46.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:02:46.199 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:02:46.199 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:02:46.200 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:02:46.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:02:46.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:02:46.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:02:46.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:02:46.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:02:46.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-03T06:02:46.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-03T06:02:46.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:02:46.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:02:46.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:02:46.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:02:46.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:02:46.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-06-03T06:02:46.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-06-03T06:02:46.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:02:46.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fbc67faf-f89f-4600-b97c-cd9fb833cc03 2024-06-03T06:02:46.201 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 fbc67faf-f89f-4600-b97c-cd9fb833cc03 2024-06-03T06:02:46.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fbc67faf-f89f-4600-b97c-cd9fb833cc03' 2024-06-03T06:02:46.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:02:46.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCGXF1mxZAEDRAAkq/HjOJ8/EJ7V1PxVyVbRA== 2024-06-03T06:02:46.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCGXF1mxZAEDRAAkq/HjOJ8/EJ7V1PxVyVbRA=="}' 2024-06-03T06:02:46.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fbc67faf-f89f-4600-b97c-cd9fb833cc03 -i td/osd-recovery-scrub/2/new.json 2024-06-03T06:02:46.587 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:02:46.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-06-03T06:02:46.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCGXF1mxZAEDRAAkq/HjOJ8/EJ7V1PxVyVbRA== --osd-uuid fbc67faf-f89f-4600-b97c-cd9fb833cc03 2024-06-03T06:02:46.629 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:46.629+0000 7ff908e03640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:46.664 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:46.664+0000 7ff908e03640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:46.665 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:46.665+0000 7ff908e03640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:46.665 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:46.665+0000 7ff908e03640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-03T06:02:46.665 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:46.666+0000 7ff908e03640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-03T06:02:46.666 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:46.666+0000 7ff908e03640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-03T06:02:46.738 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:46.738+0000 7ff908e03640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-06-03T06:02:52.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-06-03T06:02:52.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:02:52.710 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T06:02:52.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T06:02:52.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:02:53.087 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:02:53.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T06:02:53.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-06-03T06:02:53.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:02:53.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:02:53.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:02:53.115 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:53.116+0000 7f7d5e80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:53.118 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:53.118+0000 7f7d5e80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:53.119 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:53.119+0000 7f7d5e80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:53.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T06:02:53.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:02:53.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:02:53.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:02:53.381 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:02:53.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:02:53.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:02:53.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:02:53.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:02:53.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:02:53.637 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:53.637+0000 7f7d5e80f640 -1 Falling back to public interface 2024-06-03T06:02:53.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:02:54.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:02:54.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:02:54.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:02:54.676 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:02:54.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:02:54.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:02:54.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:02:55.971 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:55.971+0000 7f7d5e80f640 -1 osd.2 0 log_to_monitors true 2024-06-03T06:02:55.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:02:55.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:02:55.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:02:55.975 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:02:55.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:02:55.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:02:56.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:02:57.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:02:57.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:02:57.313 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:02:57.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:02:57.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:02:57.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:02:57.610 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2030689920,v1:127.0.0.1:6819/2030689920] [v2:127.0.0.1:6820/2030689920,v1:127.0.0.1:6821/2030689920] exists,up fbc67faf-f89f-4600-b97c-cd9fb833cc03 2024-06-03T06:02:57.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:02:57.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:02:57.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:02:57.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:02:57.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-06-03T06:02:57.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-03T06:02:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:02:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-03T06:02:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:02:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-06-03T06:02:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-03T06:02:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:02:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:02:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:02:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-06-03T06:02:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-06-03T06:02:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:02:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:02:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-03T06:02:57.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:02:57.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:02:57.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:02:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:02:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:02:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:02:57.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:02:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:02:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:02:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:02:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:02:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-03T06:02:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-03T06:02:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:02:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:02:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:02:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:02:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:02:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-06-03T06:02:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-06-03T06:02:57.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:02:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5fd360bf-0d30-4f23-9313-fd1dc5cde52a 2024-06-03T06:02:57.617 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 5fd360bf-0d30-4f23-9313-fd1dc5cde52a 2024-06-03T06:02:57.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5fd360bf-0d30-4f23-9313-fd1dc5cde52a' 2024-06-03T06:02:57.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:02:57.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRXF1m0Ji5JRAAeUcKe0wbbMJQTLYodDMPKg== 2024-06-03T06:02:57.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRXF1m0Ji5JRAAeUcKe0wbbMJQTLYodDMPKg=="}' 2024-06-03T06:02:57.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5fd360bf-0d30-4f23-9313-fd1dc5cde52a -i td/osd-recovery-scrub/3/new.json 2024-06-03T06:02:58.008 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:02:58.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-06-03T06:02:58.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCRXF1m0Ji5JRAAeUcKe0wbbMJQTLYodDMPKg== --osd-uuid 5fd360bf-0d30-4f23-9313-fd1dc5cde52a 2024-06-03T06:02:58.050 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:58.050+0000 7fddaa81b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:58.102 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:58.102+0000 7fddaa81b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:58.103 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:58.103+0000 7fddaa81b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:02:58.103 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:58.103+0000 7fddaa81b640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-03T06:02:58.103 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:58.104+0000 7fddaa81b640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-03T06:02:58.104 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:58.104+0000 7fddaa81b640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-03T06:02:58.201 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:02:58.202+0000 7fddaa81b640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-06-03T06:03:04.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-06-03T06:03:04.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:03:04.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-03T06:03:04.204 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-06-03T06:03:04.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:03:04.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-03T06:03:04.583 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-06-03T06:03:04.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-06-03T06:03:04.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:03:04.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:03:04.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:03:04.611 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:03:04.611+0000 7f9605419640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:03:04.613 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:03:04.613+0000 7f9605419640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:03:04.614 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:03:04.614+0000 7f9605419640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:03:04.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-03T06:03:04.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:03:04.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:03:04.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:03:04.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:03:04.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:03:04.874 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:03:04.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:03:04.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:03:04.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:03:05.136 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:03:05.137+0000 7f9605419640 -1 Falling back to public interface 2024-06-03T06:03:05.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:03:06.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:03:06.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:03:06.159 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:03:06.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:03:06.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:03:06.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:03:06.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:03:07.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:03:07.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:03:07.459 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:03:07.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:03:07.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:03:07.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:03:07.585 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:03:07.585+0000 7f9605419640 -1 osd.3 0 log_to_monitors true 2024-06-03T06:03:07.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:03:08.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:03:08.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:03:08.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:03:08.811 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:03:08.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:03:08.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:03:09.140 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1382864286,v1:127.0.0.1:6827/1382864286] [v2:127.0.0.1:6828/1382864286,v1:127.0.0.1:6829/1382864286] exists,up 5fd360bf-0d30-4f23-9313-fd1dc5cde52a 2024-06-03T06:03:09.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:03:09.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:03:09.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:03:09.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-06-03T06:03:09.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-06-03T06:03:09.549 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' created 2024-06-03T06:03:09.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:03:10.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-06-03T06:03:10.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:03:10.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:03:10.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:03:10.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:03:10.563 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:03:10.563 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:03:10.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:03:10.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:03:10.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:03:10.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:03:10.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:03:10.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:03:10.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:03:10.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:03:10.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:03:10.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:03:10.975 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:03:10.975 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:03:10.975 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-06-03T06:03:10.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:03:10.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:03:10.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:03:11.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-06-03T06:03:11.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-06-03T06:03:11.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-06-03T06:03:11.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:03:11.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:03:11.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-06-03T06:03:11.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-06-03T06:03:11.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2024-06-03T06:03:11.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:03:11.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:03:11.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-06-03T06:03:11.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-06-03T06:03:11.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509444' 2024-06-03T06:03:11.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:03:11.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:03:11.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345922 2024-06-03T06:03:11.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345922 2024-06-03T06:03:11.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509444 3-85899345922' 2024-06-03T06:03:11.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:03:11.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-06-03T06:03:11.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:03:11.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:03:11.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-06-03T06:03:11.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:03:11.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-06-03T06:03:11.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-06-03T06:03:11.429 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836489 2024-06-03T06:03:11.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:03:11.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836489 2024-06-03T06:03:11.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:03:12.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:03:12.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:03:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-06-03T06:03:13.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:03:13.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-06-03T06:03:13.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:03:13.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:03:13.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-06-03T06:03:13.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:03:13.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-06-03T06:03:13.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-06-03T06:03:13.021 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672967 2024-06-03T06:03:13.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:03:13.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-06-03T06:03:13.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:03:13.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-06-03T06:03:13.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:03:13.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:03:13.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-06-03T06:03:13.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:03:13.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-06-03T06:03:13.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-06-03T06:03:13.359 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509444 2024-06-03T06:03:13.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:03:13.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509444 2024-06-03T06:03:13.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:03:13.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345922 2024-06-03T06:03:13.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:03:13.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:03:13.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345922 2024-06-03T06:03:13.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:03:13.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345922 2024-06-03T06:03:13.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-06-03T06:03:13.653 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345922 2024-06-03T06:03:13.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:03:14.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345922 -lt 85899345922 2024-06-03T06:03:14.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:03:14.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:03:14.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:03:14.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:03:14.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:03:14.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:03:14.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:03:14.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:03:14.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:03:14.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:03:14.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:03:14.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:03:14.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:03:14.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:03:14.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:03:15.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:03:15.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:03:15.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:03:15.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-06-03T06:03:15.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-06-03T06:03:15.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-06-03T06:03:15.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-06-03T06:03:15.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-06-03T06:03:15.631 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:03:15.632 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:03:10.187086+0000 0'0 23:8 [1] 1 [1] 1 0'0 2024-06-03T06:03:09.492759+0000 0'0 2024-06-03T06:03:09.492759+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:04:35.796576+0000 0 0 2024-06-03T06:03:15.632 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:03:15.632 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:03:15.632 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:03:15.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.44967 bs=1M count=50 2024-06-03T06:03:15.783 INFO:tasks.workunit.client.0.smithi039.stderr:50+0 records in 2024-06-03T06:03:15.784 INFO:tasks.workunit.client.0.smithi039.stderr:50+0 records out 2024-06-03T06:03:15.784 INFO:tasks.workunit.client.0.smithi039.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.139866 s, 375 MB/s 2024-06-03T06:03:15.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-06-03T06:03:15.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:15.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.44967 2024-06-03T06:03:17.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:17.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.44967 2024-06-03T06:03:18.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:18.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.44967 2024-06-03T06:03:20.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:20.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.44967 2024-06-03T06:03:22.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:22.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.44967 2024-06-03T06:03:23.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:23.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.44967 2024-06-03T06:03:25.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:25.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.44967 2024-06-03T06:03:26.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:26.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.44967 2024-06-03T06:03:27.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:27.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.44967 2024-06-03T06:03:29.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:29.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.44967 2024-06-03T06:03:30.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:30.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.44967 2024-06-03T06:03:32.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:32.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.44967 2024-06-03T06:03:33.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:33.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.44967 2024-06-03T06:03:35.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:35.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.44967 2024-06-03T06:03:36.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:36.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.44967 2024-06-03T06:03:38.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:38.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.44967 2024-06-03T06:03:39.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:39.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.44967 2024-06-03T06:03:41.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:41.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.44967 2024-06-03T06:03:42.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:42.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.44967 2024-06-03T06:03:44.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:44.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.44967 2024-06-03T06:03:45.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:45.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.44967 2024-06-03T06:03:47.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:47.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.44967 2024-06-03T06:03:48.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:48.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.44967 2024-06-03T06:03:50.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:50.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.44967 2024-06-03T06:03:52.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:52.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.44967 2024-06-03T06:03:53.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:53.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.44967 2024-06-03T06:03:55.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:55.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.44967 2024-06-03T06:03:56.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:56.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.44967 2024-06-03T06:03:58.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:58.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.44967 2024-06-03T06:03:59.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:03:59.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.44967 2024-06-03T06:04:01.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:01.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.44967 2024-06-03T06:04:02.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:02.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.44967 2024-06-03T06:04:04.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:04.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.44967 2024-06-03T06:04:05.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:05.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.44967 2024-06-03T06:04:07.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:07.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.44967 2024-06-03T06:04:08.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:08.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.44967 2024-06-03T06:04:10.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:10.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.44967 2024-06-03T06:04:11.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:11.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.44967 2024-06-03T06:04:13.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:13.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.44967 2024-06-03T06:04:14.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:14.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.44967 2024-06-03T06:04:16.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:16.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.44967 2024-06-03T06:04:18.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:18.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.44967 2024-06-03T06:04:19.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:19.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.44967 2024-06-03T06:04:21.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:21.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.44967 2024-06-03T06:04:22.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:22.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.44967 2024-06-03T06:04:24.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:24.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.44967 2024-06-03T06:04:25.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:25.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.44967 2024-06-03T06:04:27.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:27.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.44967 2024-06-03T06:04:28.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:28.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.44967 2024-06-03T06:04:30.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:30.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.44967 2024-06-03T06:04:31.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:31.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.44967 2024-06-03T06:04:33.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:33.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.44967 2024-06-03T06:04:34.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:34.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.44967 2024-06-03T06:04:36.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:36.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.44967 2024-06-03T06:04:37.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:37.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.44967 2024-06-03T06:04:39.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:39.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.44967 2024-06-03T06:04:40.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:40.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.44967 2024-06-03T06:04:42.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:42.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.44967 2024-06-03T06:04:43.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:43.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.44967 2024-06-03T06:04:45.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:45.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.44967 2024-06-03T06:04:46.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:46.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.44967 2024-06-03T06:04:48.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:48.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.44967 2024-06-03T06:04:50.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:50.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.44967 2024-06-03T06:04:51.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:51.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.44967 2024-06-03T06:04:53.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:53.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.44967 2024-06-03T06:04:54.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:54.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.44967 2024-06-03T06:04:56.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:56.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.44967 2024-06-03T06:04:57.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:57.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.44967 2024-06-03T06:04:59.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:04:59.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.44967 2024-06-03T06:05:00.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:00.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.44967 2024-06-03T06:05:02.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:02.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.44967 2024-06-03T06:05:03.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:03.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.44967 2024-06-03T06:05:05.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:05.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.44967 2024-06-03T06:05:06.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:06.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.44967 2024-06-03T06:05:08.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:08.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.44967 2024-06-03T06:05:09.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:09.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.44967 2024-06-03T06:05:11.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:11.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.44967 2024-06-03T06:05:12.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:12.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.44967 2024-06-03T06:05:14.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:14.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.44967 2024-06-03T06:05:15.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:15.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.44967 2024-06-03T06:05:17.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:17.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.44967 2024-06-03T06:05:19.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:19.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.44967 2024-06-03T06:05:20.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:20.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.44967 2024-06-03T06:05:22.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:22.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.44967 2024-06-03T06:05:23.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:23.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.44967 2024-06-03T06:05:25.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:25.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.44967 2024-06-03T06:05:26.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:26.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.44967 2024-06-03T06:05:28.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:28.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.44967 2024-06-03T06:05:29.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:29.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.44967 2024-06-03T06:05:31.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:31.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.44967 2024-06-03T06:05:32.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:32.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.44967 2024-06-03T06:05:34.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:34.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.44967 2024-06-03T06:05:36.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:36.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.44967 2024-06-03T06:05:37.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:37.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.44967 2024-06-03T06:05:38.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:38.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.44967 2024-06-03T06:05:40.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:40.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.44967 2024-06-03T06:05:42.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:42.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.44967 2024-06-03T06:05:43.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:43.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.44967 2024-06-03T06:05:44.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:44.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.44967 2024-06-03T06:05:46.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-03T06:05:46.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.44967 2024-06-03T06:05:48.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.44967 2024-06-03T06:05:48.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-06-03T06:05:48.533 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 4 2024-06-03T06:05:48.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-06-03T06:05:48.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-06-03T06:05:48.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-06-03T06:05:48.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-06-03T06:05:48.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-06-03T06:05:48.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-06-03T06:05:48.820 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:05:48.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-06-03T06:05:50.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-06-03T06:05:50.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-06-03T06:05:50.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-06-03T06:05:50.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-06-03T06:05:50.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-06-03T06:05:50.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-06-03T06:05:50.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-06-03T06:05:51.125 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:05:51.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-06-03T06:05:53.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-06-03T06:05:53.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-06-03T06:05:53.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-06-03T06:05:53.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-06-03T06:05:53.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-06-03T06:05:53.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-06-03T06:05:53.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-06-03T06:05:53.586 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:05:53.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-06-03T06:05:55.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2024-06-03T06:05:55.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2024-06-03T06:05:55.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2024-06-03T06:05:55.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-06-03T06:05:55.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-06-03T06:05:55.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-06-03T06:05:55.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-06-03T06:05:56.019 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:05:56.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-06-03T06:05:58.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 3 -eq 10 2024-06-03T06:05:58.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 3 + 1 2024-06-03T06:05:58.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=4 2024-06-03T06:05:58.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-06-03T06:05:58.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-06-03T06:05:58.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-06-03T06:05:58.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-06-03T06:05:58.322 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:05:58.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-06-03T06:05:58.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-06-03T06:05:58.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-06-03T06:05:58.833 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:05:58.833 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 297 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-06-03T06:05:51.362760+0000 24'1300 27:1327 [1,0,2,3] 1 [1,0] 1 0'0 2024-06-03T06:03:09.492759+0000 0'0 2024-06-03T06:03:09.492759+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:05:58.833 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:05:58.833 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:05:58.833 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:05:58.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-06-03T06:06:08.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-06-03T06:06:08.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:06:08.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:06:08.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:06:08.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:06:08.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:06:14.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:06:14.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-06-03T06:06:14.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress. Only high priority scrubs allowed.' 2024-06-03T06:06:14.163 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-06-03T06:06:14.164 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-06-03T06:06:14.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:06:14.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-06-03T06:06:14.168 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:05:55.168+0000 7f8e59f46640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.168 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:01.072+0000 7f8e59f46640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.168 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:03.012+0000 7f8e59f46640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.168 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:05.059+0000 7f8e59f46640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.168 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:07.087+0000 7f8e59f46640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.168 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:12.110+0000 7f8e59f46640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.169 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:13.109+0000 7f8e59f46640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:06:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-06-03T06:06:14.180 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:05:56.953+0000 7fa286aa4640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.181 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:03.054+0000 7fa286aa4640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.181 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:05.003+0000 7fa286aa4640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.181 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:06.988+0000 7fa286aa4640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.181 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:10.985+0000 7fa286aa4640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:06:14.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-06-03T06:06:14.183 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:05:51.898+0000 7f7d571db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.183 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:05:56.897+0000 7f7d571db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.184 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:05:58.899+0000 7f7d571db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.184 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:01.902+0000 7f7d571db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.184 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:05.912+0000 7f7d571db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.184 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:09.948+0000 7f7d571db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.184 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:11.891+0000 7f7d571db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:06:14.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-06-03T06:06:14.186 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:05:56.951+0000 7f95fde14640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.186 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:05:57.961+0000 7f95fde14640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.186 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:01.981+0000 7f95fde14640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.186 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:03.025+0000 7f95fde14640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.187 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:04.060+0000 7f95fde14640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.187 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:08.077+0000 7f95fde14640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.187 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:06:12.143+0000 7f95fde14640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-03T06:06:14.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-06-03T06:06:14.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-06-03T06:06:14.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-06-03T06:06:14.188 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-06-03T06:06:14.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-06-03T06:06:14.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:06:14.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-06-03T06:06:14.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-06-03T06:06:14.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-06-03T06:06:14.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-06-03T06:06:14.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:06:14.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-06-03T06:06:14.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-06-03T06:06:14.205 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-06-03T06:06:14.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-06-03T06:06:14.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:06:14.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-06-03T06:06:14.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-06-03T06:06:14.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-06-03T06:06:14.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-06-03T06:06:14.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:06:14.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-06-03T06:06:14.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-06-03T06:06:14.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-06-03T06:06:14.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-06-03T06:06:14.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-06-03T06:06:14.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-06-03T06:06:14.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-06-03T06:06:14.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-06-03T06:06:14.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:06:14.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-06-03T06:06:14.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:06:14.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:06:14.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:06:14.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:06:14.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:06:14.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:06:14.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:06:14.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:06:14.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:06:14.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:06:14.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:06:14.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:06:14.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:06:14.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:06:14.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:06:14.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:06:14.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:06:14.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:06:14.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-06-03T06:06:14.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:06:14.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:06:14.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:06:14.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44967 2024-06-03T06:06:14.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:06:14.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:06:14.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-06-03T06:06:14.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-06-03T06:06:14.252 INFO:tasks.workunit.client.0.smithi039.stdout:TEST PASSED 2024-06-03T06:06:14.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-06-03T06:06:14.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-06-03T06:06:14.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-06-03T06:06:14.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-06-03T06:06:14.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2024-06-03T06:06:14.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.44967 2024-06-03T06:06:14.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2024-06-03T06:06:14.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2024-06-03T06:06:14.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2024-06-03T06:06:14.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2024-06-03T06:06:14.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-06-03T06:06:14.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-06-03T06:06:14.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-06-03T06:06:14.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:06:14.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-06-03T06:06:14.255 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:06:14.255 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:06:14.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:06:14.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:06:14.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:06:14.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:06:14.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:06:14.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:06:14.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:06:14.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:06:14.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:06:14.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:06:14.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:06:14.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:06:14.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:06:14.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:06:14.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:06:14.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:06:14.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-06-03T06:06:14.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:06:14.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:06:14.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:06:14.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44967 2024-06-03T06:06:14.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:06:14.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:06:14.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-06-03T06:06:14.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:06:14.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:06:14.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:06:14.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.44967 2024-06-03T06:06:14.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:06:14.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T06:06:14.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:06:14.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-06-03T06:06:14.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T06:06:14.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-06-03T06:06:14.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T06:06:14.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T06:06:14.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T06:06:14.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-06-03T06:06:14.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T06:06:14.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T06:06:14.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:06:14.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:06:14.640 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:06:14.640 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:06:14.640 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:06:14.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:06:14.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T06:06:14.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T06:06:14.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T06:06:14.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:06:14.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:06:14.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T06:06:14.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T06:06:14.835 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:06:14.835 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:06:14.835 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:06:14.835 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:06:14.836 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:06:14.836 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:06:14.836 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44967/ceph-mon.a.asok 2024-06-03T06:06:14.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:06:14.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44967/ceph-mon.a.asok config get fsid 2024-06-03T06:06:14.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T06:06:14.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:06:14.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:06:14.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T06:06:14.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T06:06:14.926 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:06:14.926 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:06:14.926 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:06:14.927 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:06:14.927 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:06:14.927 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:06:14.927 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44967/ceph-mon.a.asok 2024-06-03T06:06:14.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:06:14.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44967/ceph-mon.a.asok config get mon_host 2024-06-03T06:06:15.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2024-06-03T06:06:15.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-06-03T06:06:15.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T06:06:15.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T06:06:15.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T06:06:15.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-06-03T06:06:15.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T06:06:15.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T06:06:15.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:06:15.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:06:15.826 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:06:15.826 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:06:15.826 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:06:15.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:06:15.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:06:15.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:06:15.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-06-03T06:06:15.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-06-03T06:06:15.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-06-03T06:06:15.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-06-03T06:06:15.857 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2024-06-03T06:06:15.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2024-06-03T06:06:15.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:06:15.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:06:15.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-03T06:06:15.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:06:15.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T06:06:15.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:06:15.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-06-03T06:06:15.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-03T06:06:15.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:06:15.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:06:15.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:06:15.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-06-03T06:06:15.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-06-03T06:06:15.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:06:15.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:06:15.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-03T06:06:15.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:06:15.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:06:15.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:06:15.864 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:06:15.864 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:06:15.864 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:06:15.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:06:15.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:06:15.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:06:15.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:06:15.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:06:15.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-03T06:06:15.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-03T06:06:15.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:06:15.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:06:15.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:06:15.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:06:15.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:06:15.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-03T06:06:15.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-06-03T06:06:15.865 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:06:15.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=82a7413e-cfd7-4b58-8c6b-dc9ab18a4374 2024-06-03T06:06:15.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 82a7413e-cfd7-4b58-8c6b-dc9ab18a4374' 2024-06-03T06:06:15.866 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 82a7413e-cfd7-4b58-8c6b-dc9ab18a4374 2024-06-03T06:06:15.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:06:15.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXXV1mAsuVNBAAxWWOPjn6WB384RUrDC9PBA== 2024-06-03T06:06:15.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXXV1mAsuVNBAAxWWOPjn6WB384RUrDC9PBA=="}' 2024-06-03T06:06:15.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 82a7413e-cfd7-4b58-8c6b-dc9ab18a4374 -i td/osd-recovery-scrub/0/new.json 2024-06-03T06:06:16.131 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:06:16.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-06-03T06:06:16.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBXXV1mAsuVNBAAxWWOPjn6WB384RUrDC9PBA== --osd-uuid 82a7413e-cfd7-4b58-8c6b-dc9ab18a4374 2024-06-03T06:06:16.176 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:16.176+0000 7f9f51a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:16.182 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:16.182+0000 7f9f51a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:16.183 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:16.184+0000 7f9f51a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:16.184 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:16.184+0000 7f9f51a15640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-03T06:06:16.184 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:16.184+0000 7f9f51a15640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-03T06:06:16.184 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:16.184+0000 7f9f51a15640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-03T06:06:16.283 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:16.283+0000 7f9f51a15640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-06-03T06:06:21.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-06-03T06:06:21.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:06:21.832 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:06:21.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T06:06:21.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:06:22.213 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:06:22.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T06:06:22.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:06:22.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:06:22.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:06:22.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:06:22.245 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:22.245+0000 7f6a7c215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:22.257 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:22.257+0000 7f6a7c215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:22.258 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:22.258+0000 7f6a7c215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:22.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T06:06:22.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:06:22.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:06:22.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:06:22.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:06:22.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:22.530 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:06:22.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:06:22.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:22.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:06:22.789 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:22.789+0000 7f6a7c215640 -1 Falling back to public interface 2024-06-03T06:06:22.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:06:23.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:06:23.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:23.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:06:23.824 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:06:23.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:23.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:06:24.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:06:25.025 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:25.025+0000 7f6a7c215640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:06:25.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:06:25.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:25.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:06:25.124 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:06:25.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:25.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:06:25.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:06:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:06:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:26.495 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:06:26.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:06:26.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:26.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:06:26.791 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2171431283,v1:127.0.0.1:6803/2171431283] [v2:127.0.0.1:6804/2171431283,v1:127.0.0.1:6805/2171431283] exists,up 82a7413e-cfd7-4b58-8c6b-dc9ab18a4374 2024-06-03T06:06:26.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:06:26.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:06:26.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:06:26.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:06:26.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:06:26.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-03T06:06:26.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:06:26.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T06:06:26.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:06:26.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-06-03T06:06:26.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-03T06:06:26.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:06:26.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:06:26.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:06:26.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-06-03T06:06:26.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-06-03T06:06:26.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:06:26.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:06:26.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-03T06:06:26.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:06:26.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:06:26.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:06:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:06:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:06:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:06:26.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:06:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:06:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:06:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:06:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:06:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-03T06:06:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-03T06:06:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:06:26.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:06:26.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:06:26.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:06:26.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:06:26.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-03T06:06:26.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-06-03T06:06:26.798 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:06:26.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ad33e853-118b-45ca-99ca-3755b7f4d31a 2024-06-03T06:06:26.798 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 ad33e853-118b-45ca-99ca-3755b7f4d31a 2024-06-03T06:06:26.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ad33e853-118b-45ca-99ca-3755b7f4d31a' 2024-06-03T06:06:26.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:06:26.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBiXV1m3qyEMBAAiwUpFjqOpMMplfCRJzey2A== 2024-06-03T06:06:26.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBiXV1m3qyEMBAAiwUpFjqOpMMplfCRJzey2A=="}' 2024-06-03T06:06:26.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ad33e853-118b-45ca-99ca-3755b7f4d31a -i td/osd-recovery-scrub/1/new.json 2024-06-03T06:06:27.155 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:06:27.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-06-03T06:06:27.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBiXV1m3qyEMBAAiwUpFjqOpMMplfCRJzey2A== --osd-uuid ad33e853-118b-45ca-99ca-3755b7f4d31a 2024-06-03T06:06:27.200 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:27.200+0000 7f526c91f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:27.222 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:27.223+0000 7f526c91f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:27.223 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:27.224+0000 7f526c91f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:27.224 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:27.224+0000 7f526c91f640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-03T06:06:27.224 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:27.224+0000 7f526c91f640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-03T06:06:27.224 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:27.224+0000 7f526c91f640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-03T06:06:27.289 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:27.289+0000 7f526c91f640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-06-03T06:06:32.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-06-03T06:06:32.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:06:32.481 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:06:32.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T06:06:32.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:06:32.864 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:06:32.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T06:06:32.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:06:32.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:06:32.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:06:32.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:06:32.897 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:32.897+0000 7fb98d708640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:32.899+0000 7fb98d708640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:32.900 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:32.900+0000 7fb98d708640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:33.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T06:06:33.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:06:33.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:06:33.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:06:33.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:06:33.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:33.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:06:33.196 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:06:33.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:33.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:06:33.415 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:33.415+0000 7fb98d708640 -1 Falling back to public interface 2024-06-03T06:06:33.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:06:34.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:06:34.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:34.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:06:34.523 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:06:34.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:34.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:06:34.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:06:35.757 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:35.757+0000 7fb98d708640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:06:35.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:06:35.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:35.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:06:35.822 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:06:35.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:35.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:06:36.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:06:37.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:06:37.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:37.176 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:06:37.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:06:37.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:37.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:06:37.474 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/124093390,v1:127.0.0.1:6811/124093390] [v2:127.0.0.1:6812/124093390,v1:127.0.0.1:6813/124093390] exists,up ad33e853-118b-45ca-99ca-3755b7f4d31a 2024-06-03T06:06:37.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:06:37.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:06:37.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:06:37.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:06:37.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:06:37.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-03T06:06:37.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:06:37.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T06:06:37.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:06:37.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-06-03T06:06:37.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-03T06:06:37.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:06:37.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:06:37.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:06:37.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-06-03T06:06:37.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-06-03T06:06:37.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:06:37.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:06:37.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-03T06:06:37.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:06:37.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:06:37.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:06:37.481 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:06:37.481 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:06:37.481 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:06:37.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:06:37.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:06:37.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:06:37.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:06:37.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:06:37.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-03T06:06:37.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-03T06:06:37.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:06:37.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:06:37.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:06:37.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:06:37.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:06:37.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-03T06:06:37.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-06-03T06:06:37.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:06:37.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=45f8e467-013f-4a59-ae3d-d9cc77dfd099 2024-06-03T06:06:37.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 45f8e467-013f-4a59-ae3d-d9cc77dfd099' 2024-06-03T06:06:37.482 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 45f8e467-013f-4a59-ae3d-d9cc77dfd099 2024-06-03T06:06:37.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:06:37.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBtXV1m2FGuHRAAAcneUvtBZVRLR+OrffeUpg== 2024-06-03T06:06:37.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBtXV1m2FGuHRAAAcneUvtBZVRLR+OrffeUpg=="}' 2024-06-03T06:06:37.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 45f8e467-013f-4a59-ae3d-d9cc77dfd099 -i td/osd-recovery-scrub/2/new.json 2024-06-03T06:06:37.877 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:06:37.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-06-03T06:06:37.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBtXV1m2FGuHRAAAcneUvtBZVRLR+OrffeUpg== --osd-uuid 45f8e467-013f-4a59-ae3d-d9cc77dfd099 2024-06-03T06:06:37.923 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:37.923+0000 7fb1dc294640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:37.961 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:37.961+0000 7fb1dc294640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:37.962 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:37.963+0000 7fb1dc294640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:37.963 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:37.963+0000 7fb1dc294640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-03T06:06:37.963 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:37.963+0000 7fb1dc294640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-03T06:06:37.963 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:37.963+0000 7fb1dc294640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-03T06:06:38.036 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:38.037+0000 7fb1dc294640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-06-03T06:06:43.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-06-03T06:06:43.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:06:43.474 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T06:06:43.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T06:06:43.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:06:43.855 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:06:43.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T06:06:43.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:06:43.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:06:43.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:06:43.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:06:43.887 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:43.887+0000 7fb9f960f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:43.889 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:43.889+0000 7fb9f960f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:43.890 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:43.890+0000 7fb9f960f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:44.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T06:06:44.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:06:44.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:06:44.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:06:44.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:06:44.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:44.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:06:44.183 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:06:44.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:44.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:06:44.410 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:44.410+0000 7fb9f960f640 -1 Falling back to public interface 2024-06-03T06:06:44.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:06:45.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:06:45.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:45.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:06:45.470 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:06:45.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:45.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:06:45.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:06:46.757 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:46.757+0000 7fb9f960f640 -1 osd.2 0 log_to_monitors true 2024-06-03T06:06:46.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:06:46.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:46.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:06:46.774 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:06:46.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:46.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:06:47.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:06:48.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:06:48.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:48.085 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:06:48.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:06:48.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:48.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:06:48.382 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2580935908,v1:127.0.0.1:6819/2580935908] [v2:127.0.0.1:6820/2580935908,v1:127.0.0.1:6821/2580935908] exists,up 45f8e467-013f-4a59-ae3d-d9cc77dfd099 2024-06-03T06:06:48.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:06:48.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:06:48.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:06:48.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:06:48.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:06:48.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-03T06:06:48.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:06:48.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-03T06:06:48.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:06:48.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-06-03T06:06:48.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-03T06:06:48.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:06:48.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:06:48.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:06:48.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-06-03T06:06:48.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-06-03T06:06:48.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:06:48.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:06:48.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-03T06:06:48.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:06:48.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:06:48.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:06:48.388 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:06:48.388 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:06:48.388 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:06:48.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:06:48.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:06:48.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:06:48.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:06:48.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:06:48.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-03T06:06:48.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-03T06:06:48.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:06:48.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:06:48.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:06:48.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:06:48.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:06:48.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-03T06:06:48.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-06-03T06:06:48.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:06:48.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae6c8e88-978b-4aae-8e33-4f2b2398eb8e 2024-06-03T06:06:48.390 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 ae6c8e88-978b-4aae-8e33-4f2b2398eb8e 2024-06-03T06:06:48.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 ae6c8e88-978b-4aae-8e33-4f2b2398eb8e' 2024-06-03T06:06:48.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:06:48.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB4XV1mTtcxGBAA5i0EWyF4uQDyUFQBKFF7kw== 2024-06-03T06:06:48.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB4XV1mTtcxGBAA5i0EWyF4uQDyUFQBKFF7kw=="}' 2024-06-03T06:06:48.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae6c8e88-978b-4aae-8e33-4f2b2398eb8e -i td/osd-recovery-scrub/3/new.json 2024-06-03T06:06:48.768 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:06:48.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-06-03T06:06:48.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQB4XV1mTtcxGBAA5i0EWyF4uQDyUFQBKFF7kw== --osd-uuid ae6c8e88-978b-4aae-8e33-4f2b2398eb8e 2024-06-03T06:06:48.814 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:48.814+0000 7fca44c0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:48.830 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:48.827+0000 7fca44c0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:48.830 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:48.828+0000 7fca44c0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:48.830 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:48.829+0000 7fca44c0c640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-03T06:06:48.830 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:48.829+0000 7fca44c0c640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-03T06:06:48.830 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:48.829+0000 7fca44c0c640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-03T06:06:48.886 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:48.886+0000 7fca44c0c640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-06-03T06:06:54.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-06-03T06:06:54.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:06:54.349 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-06-03T06:06:54.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-03T06:06:54.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:06:54.723 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-06-03T06:06:54.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-03T06:06:54.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:06:54.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:06:54.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:06:54.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:06:54.755 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:54.754+0000 7ff50ac12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:54.756 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:54.756+0000 7ff50ac12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:54.757 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:54.757+0000 7ff50ac12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:55.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-03T06:06:55.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:06:55.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:06:55.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:06:55.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:06:55.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:55.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:06:55.012 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:06:55.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:55.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:06:55.280 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:55.280+0000 7ff50ac12640 -1 Falling back to public interface 2024-06-03T06:06:55.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:06:56.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:06:56.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:56.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:06:56.311 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:06:56.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:56.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:06:56.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:06:57.607 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:57.606+0000 7ff50ac12640 -1 osd.3 0 log_to_monitors true 2024-06-03T06:06:57.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:06:57.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:57.612 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:06:57.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:06:57.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:57.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:06:57.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:06:58.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:06:58.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:06:58.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:06:58.983 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:06:58.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:06:58.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:06:59.282 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3046533658,v1:127.0.0.1:6827/3046533658] [v2:127.0.0.1:6828/3046533658,v1:127.0.0.1:6829/3046533658] exists,up ae6c8e88-978b-4aae-8e33-4f2b2398eb8e 2024-06-03T06:06:59.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:06:59.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:06:59.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:06:59.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:06:59.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:06:59.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-03T06:06:59.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:06:59.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-06-03T06:06:59.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:06:59.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2024-06-03T06:06:59.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-03T06:06:59.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:06:59.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:06:59.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:06:59.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2024-06-03T06:06:59.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2024-06-03T06:06:59.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:06:59.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:06:59.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-03T06:06:59.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:06:59.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:06:59.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:06:59.288 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:06:59.288 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:06:59.288 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:06:59.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:06:59.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:06:59.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:06:59.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:06:59.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:06:59.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-03T06:06:59.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-03T06:06:59.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:06:59.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:06:59.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:06:59.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:06:59.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:06:59.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-03T06:06:59.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2024-06-03T06:06:59.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:06:59.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c8cda1a2-8906-4bd8-a03b-000711ed6c79 2024-06-03T06:06:59.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 c8cda1a2-8906-4bd8-a03b-000711ed6c79' 2024-06-03T06:06:59.290 INFO:tasks.workunit.client.0.smithi039.stdout:add osd4 c8cda1a2-8906-4bd8-a03b-000711ed6c79 2024-06-03T06:06:59.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:06:59.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCDXV1my5FAEhAA6awf8uowtDGOQQbFE0aA2A== 2024-06-03T06:06:59.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCDXV1my5FAEhAA6awf8uowtDGOQQbFE0aA2A=="}' 2024-06-03T06:06:59.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c8cda1a2-8906-4bd8-a03b-000711ed6c79 -i td/osd-recovery-scrub/4/new.json 2024-06-03T06:06:59.670 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:06:59.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2024-06-03T06:06:59.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCDXV1my5FAEhAA6awf8uowtDGOQQbFE0aA2A== --osd-uuid c8cda1a2-8906-4bd8-a03b-000711ed6c79 2024-06-03T06:06:59.715 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:59.716+0000 7fe384630640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:59.746 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:59.746+0000 7fe384630640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:59.747 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:59.747+0000 7fe384630640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:06:59.747 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:59.748+0000 7fe384630640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-06-03T06:06:59.748 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:59.748+0000 7fe384630640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-06-03T06:06:59.748 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:59.748+0000 7fe384630640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-06-03T06:06:59.822 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:06:59.823+0000 7fe384630640 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2024-06-03T06:07:05.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2024-06-03T06:07:05.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:07:05.565 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd4 key to auth repository 2024-06-03T06:07:05.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-06-03T06:07:05.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:07:05.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-06-03T06:07:05.943 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.4 2024-06-03T06:07:05.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:07:05.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:07:05.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:07:05.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:07:05.974 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:05.974+0000 7f7d40133640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:05.976 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:05.977+0000 7f7d40133640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:05.977 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:05.978+0000 7f7d40133640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:06.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-06-03T06:07:06.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:07:06.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-06-03T06:07:06.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:07:06.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:07:06.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:06.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:07:06.237 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:07:06.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:06.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:07:06.498 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:06.498+0000 7f7d40133640 -1 Falling back to public interface 2024-06-03T06:07:06.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:07:07.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:07:07.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:07.549 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:07:07.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:07:07.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:07.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:07:07.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:07:08.844 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:08.844+0000 7f7d40133640 -1 osd.4 0 log_to_monitors true 2024-06-03T06:07:08.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:07:08.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:08.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:07:08.848 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:07:08.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:08.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:07:09.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:07:10.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:07:10.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:10.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:07:10.169 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:07:10.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:10.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:07:10.471 INFO:tasks.workunit.client.0.smithi039.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2082054296,v1:127.0.0.1:6835/2082054296] [v2:127.0.0.1:6836/2082054296,v1:127.0.0.1:6837/2082054296] exists,up c8cda1a2-8906-4bd8-a03b-000711ed6c79 2024-06-03T06:07:10.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:07:10.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:07:10.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:07:10.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:07:10.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:07:10.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-03T06:07:10.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:07:10.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-06-03T06:07:10.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:07:10.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2024-06-03T06:07:10.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-03T06:07:10.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:07:10.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:07:10.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:07:10.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2024-06-03T06:07:10.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2024-06-03T06:07:10.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:07:10.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:07:10.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-03T06:07:10.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:07:10.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:07:10.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:07:10.476 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:07:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:07:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:07:10.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:07:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:07:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:07:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:07:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:07:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-03T06:07:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-03T06:07:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:07:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:07:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:07:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:07:10.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:07:10.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-03T06:07:10.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2024-06-03T06:07:10.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:07:10.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ff4d6c92-5db7-4ba7-b1e8-d106586d8120 2024-06-03T06:07:10.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 ff4d6c92-5db7-4ba7-b1e8-d106586d8120' 2024-06-03T06:07:10.479 INFO:tasks.workunit.client.0.smithi039.stdout:add osd5 ff4d6c92-5db7-4ba7-b1e8-d106586d8120 2024-06-03T06:07:10.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:07:10.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCOXV1mSTB0HRAA/jcyjDaCc0d3CjyTqhtrkQ== 2024-06-03T06:07:10.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCOXV1mSTB0HRAA/jcyjDaCc0d3CjyTqhtrkQ=="}' 2024-06-03T06:07:10.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ff4d6c92-5db7-4ba7-b1e8-d106586d8120 -i td/osd-recovery-scrub/5/new.json 2024-06-03T06:07:10.842 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:07:10.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2024-06-03T06:07:10.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCOXV1mSTB0HRAA/jcyjDaCc0d3CjyTqhtrkQ== --osd-uuid ff4d6c92-5db7-4ba7-b1e8-d106586d8120 2024-06-03T06:07:10.888 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:10.888+0000 7fbc899a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:10.909 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:10.910+0000 7fbc899a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:10.910 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:10.911+0000 7fbc899a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:10.911 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:10.911+0000 7fbc899a5640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-06-03T06:07:10.911 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:10.911+0000 7fbc899a5640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-06-03T06:07:10.911 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:10.911+0000 7fbc899a5640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-06-03T06:07:10.984 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:10.985+0000 7fbc899a5640 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2024-06-03T06:07:16.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2024-06-03T06:07:16.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:07:16.416 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd5 key to auth repository 2024-06-03T06:07:16.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-06-03T06:07:16.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:07:16.792 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.5 2024-06-03T06:07:16.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-06-03T06:07:16.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:07:16.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:07:16.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:07:16.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:07:16.824 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:16.824+0000 7fb242985640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:16.826 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:16.826+0000 7fb242985640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:16.827 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:16.827+0000 7fb242985640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:17.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-06-03T06:07:17.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:07:17.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-03T06:07:17.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:07:17.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:07:17.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:17.099 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:07:17.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:07:17.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:17.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:07:17.352 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:17.352+0000 7fb242985640 -1 Falling back to public interface 2024-06-03T06:07:17.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:07:18.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:07:18.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:18.393 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:07:18.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:07:18.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:18.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:07:18.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:07:19.679 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:19.680+0000 7fb242985640 -1 osd.5 0 log_to_monitors true 2024-06-03T06:07:19.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:07:19.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:19.697 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:07:19.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:07:19.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:19.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:07:20.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:07:21.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:07:21.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:21.080 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:07:21.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:07:21.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:21.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:07:21.380 INFO:tasks.workunit.client.0.smithi039.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/4278704106,v1:127.0.0.1:6843/4278704106] [v2:127.0.0.1:6844/4278704106,v1:127.0.0.1:6845/4278704106] exists,up ff4d6c92-5db7-4ba7-b1e8-d106586d8120 2024-06-03T06:07:21.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:07:21.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:07:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:07:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:07:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:07:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-03T06:07:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:07:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-06-03T06:07:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:07:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2024-06-03T06:07:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-03T06:07:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:07:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:07:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:07:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2024-06-03T06:07:21.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2024-06-03T06:07:21.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:07:21.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:07:21.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-03T06:07:21.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:07:21.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:07:21.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:07:21.385 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:07:21.385 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:07:21.385 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:07:21.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:07:21.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:07:21.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:07:21.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:07:21.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:07:21.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-03T06:07:21.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-03T06:07:21.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:07:21.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:07:21.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:07:21.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:07:21.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:07:21.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-03T06:07:21.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2024-06-03T06:07:21.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:07:21.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dc3ba592-8929-4ea2-8912-ceb2f61e24bd 2024-06-03T06:07:21.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 dc3ba592-8929-4ea2-8912-ceb2f61e24bd' 2024-06-03T06:07:21.387 INFO:tasks.workunit.client.0.smithi039.stdout:add osd6 dc3ba592-8929-4ea2-8912-ceb2f61e24bd 2024-06-03T06:07:21.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:07:21.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCZXV1mRSL7FxAAyI0FnQmzC8i/Ds0bB9MdsQ== 2024-06-03T06:07:21.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCZXV1mRSL7FxAAyI0FnQmzC8i/Ds0bB9MdsQ=="}' 2024-06-03T06:07:21.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dc3ba592-8929-4ea2-8912-ceb2f61e24bd -i td/osd-recovery-scrub/6/new.json 2024-06-03T06:07:21.792 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:07:21.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2024-06-03T06:07:21.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCZXV1mRSL7FxAAyI0FnQmzC8i/Ds0bB9MdsQ== --osd-uuid dc3ba592-8929-4ea2-8912-ceb2f61e24bd 2024-06-03T06:07:21.837 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:21.838+0000 7feb32e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:21.878 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:21.879+0000 7feb32e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:21.879 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:21.880+0000 7feb32e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:21.880 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:21.880+0000 7feb32e0c640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-06-03T06:07:21.880 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:21.880+0000 7feb32e0c640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-06-03T06:07:21.880 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:21.880+0000 7feb32e0c640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-06-03T06:07:21.951 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:21.951+0000 7feb32e0c640 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2024-06-03T06:07:27.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2024-06-03T06:07:27.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:07:27.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-06-03T06:07:27.989 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd6 key to auth repository 2024-06-03T06:07:27.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:07:28.366 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.6 2024-06-03T06:07:28.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-06-03T06:07:28.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:07:28.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:07:28.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:07:28.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:07:28.400 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:28.400+0000 7f4490414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:28.402 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:28.402+0000 7f4490414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:28.403 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:28.403+0000 7f4490414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:28.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-06-03T06:07:28.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:07:28.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-06-03T06:07:28.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:07:28.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:07:28.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:28.656 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:07:28.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:07:28.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:28.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:07:28.917 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:28.918+0000 7f4490414640 -1 Falling back to public interface 2024-06-03T06:07:28.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:07:29.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:07:29.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:29.966 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:07:29.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:07:29.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:29.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:07:30.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:07:31.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:07:31.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:31.268 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:07:31.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:07:31.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:31.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:07:31.364 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:31.364+0000 7f4490414640 -1 osd.6 0 log_to_monitors true 2024-06-03T06:07:31.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:07:32.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:07:32.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:32.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:07:32.574 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:07:32.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:32.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:07:32.895 INFO:tasks.workunit.client.0.smithi039.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/4095701625,v1:127.0.0.1:6851/4095701625] [v2:127.0.0.1:6852/4095701625,v1:127.0.0.1:6853/4095701625] exists,up dc3ba592-8929-4ea2-8912-ceb2f61e24bd 2024-06-03T06:07:32.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:07:32.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:07:32.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:07:32.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:07:32.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:07:32.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-03T06:07:32.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:07:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-06-03T06:07:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:07:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2024-06-03T06:07:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-03T06:07:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:07:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:07:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:07:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2024-06-03T06:07:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2024-06-03T06:07:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:07:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:07:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-03T06:07:32.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:07:32.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:07:32.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:07:32.900 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:07:32.900 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:07:32.900 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:07:32.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:07:32.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' 2024-06-03T06:07:32.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:07:32.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:07:32.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:07:32.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-03T06:07:32.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-03T06:07:32.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:07:32.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:07:32.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:07:32.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:07:32.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:07:32.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-03T06:07:32.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2024-06-03T06:07:32.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:07:32.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5cd36ab8-16af-4883-93f8-5da6331132fe 2024-06-03T06:07:32.902 INFO:tasks.workunit.client.0.smithi039.stdout:add osd7 5cd36ab8-16af-4883-93f8-5da6331132fe 2024-06-03T06:07:32.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 5cd36ab8-16af-4883-93f8-5da6331132fe' 2024-06-03T06:07:32.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:07:32.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCkXV1mB1euNhAA8wd/dxN3tDjvcDTTBjUpdA== 2024-06-03T06:07:32.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCkXV1mB1euNhAA8wd/dxN3tDjvcDTTBjUpdA=="}' 2024-06-03T06:07:32.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5cd36ab8-16af-4883-93f8-5da6331132fe -i td/osd-recovery-scrub/7/new.json 2024-06-03T06:07:33.262 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:07:33.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2024-06-03T06:07:33.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCkXV1mB1euNhAA8wd/dxN3tDjvcDTTBjUpdA== --osd-uuid 5cd36ab8-16af-4883-93f8-5da6331132fe 2024-06-03T06:07:33.307 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:33.308+0000 7f549c62c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:33.346 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:33.346+0000 7f549c62c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:33.347 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:33.347+0000 7f549c62c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:33.348 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:33.348+0000 7f549c62c640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-06-03T06:07:33.348 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:33.348+0000 7f549c62c640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-06-03T06:07:33.348 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:33.348+0000 7f549c62c640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-06-03T06:07:33.421 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:33.421+0000 7f549c62c640 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2024-06-03T06:07:39.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2024-06-03T06:07:39.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:07:39.363 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd7 key to auth repository 2024-06-03T06:07:39.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-06-03T06:07:39.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:07:39.739 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.7 2024-06-03T06:07:39.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-06-03T06:07:39.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=9862db72-b4fc-4693-85b8-78cfd05cc927 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-03T06:07:39.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:07:39.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:07:39.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:07:39.772 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:39.772+0000 7f46ec38e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:39.774 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:39.774+0000 7f46ec38e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:39.775 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:39.775+0000 7f46ec38e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:07:40.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-06-03T06:07:40.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:07:40.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-06-03T06:07:40.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:07:40.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:07:40.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:40.031 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:07:40.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:07:40.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:40.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:07:40.309 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:40.310+0000 7f46ec38e640 -1 Falling back to public interface 2024-06-03T06:07:40.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:07:41.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:07:41.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:41.323 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:07:41.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:07:41.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:41.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:07:41.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:07:42.592 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:07:42.592+0000 7f46ec38e640 -1 osd.7 0 log_to_monitors true 2024-06-03T06:07:42.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:07:42.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:42.621 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:07:42.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:07:42.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:42.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:07:42.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:07:43.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:07:43.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:07:43.989 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:07:43.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:07:43.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:07:43.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:07:44.281 INFO:tasks.workunit.client.0.smithi039.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/80501616,v1:127.0.0.1:6859/80501616] [v2:127.0.0.1:6860/80501616,v1:127.0.0.1:6861/80501616] exists,up 5cd36ab8-16af-4883-93f8-5da6331132fe 2024-06-03T06:07:44.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:07:44.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:07:44.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:07:44.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2024-06-03T06:07:44.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2024-06-03T06:07:44.663 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' created 2024-06-03T06:07:44.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:07:45.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2024-06-03T06:07:45.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:07:45.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:07:45.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:07:45.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:07:45.677 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:07:45.678 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:07:45.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:07:45.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:07:45.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:07:45.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:07:45.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:07:45.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:07:45.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:07:45.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:07:45.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:07:46.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:07:46.213 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:07:46.213 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:07:46.213 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:07:46.213 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-06-03T06:07:46.213 INFO:tasks.workunit.client.0.smithi039.stderr:5 2024-06-03T06:07:46.213 INFO:tasks.workunit.client.0.smithi039.stderr:6 2024-06-03T06:07:46.213 INFO:tasks.workunit.client.0.smithi039.stderr:7' 2024-06-03T06:07:46.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:07:46.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:07:46.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:07:46.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-06-03T06:07:46.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-06-03T06:07:46.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-06-03T06:07:46.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:07:46.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:07:46.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-06-03T06:07:46.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-06-03T06:07:46.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975' 2024-06-03T06:07:46.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:07:46.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:07:46.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-06-03T06:07:46.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-06-03T06:07:46.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453' 2024-06-03T06:07:46.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:07:46.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:07:47.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345931 2024-06-03T06:07:47.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345931 2024-06-03T06:07:47.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453 3-85899345931' 2024-06-03T06:07:47.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:07:47.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:07:47.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182409 2024-06-03T06:07:47.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182409 2024-06-03T06:07:47.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453 3-85899345931 4-107374182409' 2024-06-03T06:07:47.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:07:47.255 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:07:47.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-06-03T06:07:47.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-06-03T06:07:47.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453 3-85899345931 4-107374182409 5-128849018887' 2024-06-03T06:07:47.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:07:47.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:07:47.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855364 2024-06-03T06:07:47.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855364 2024-06-03T06:07:47.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453 3-85899345931 4-107374182409 5-128849018887 6-150323855364' 2024-06-03T06:07:47.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:07:47.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:07:47.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-06-03T06:07:47.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-06-03T06:07:47.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453 3-85899345931 4-107374182409 5-128849018887 6-150323855364 7-171798691842' 2024-06-03T06:07:47.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:07:47.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-06-03T06:07:47.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:07:47.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:07:47.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-06-03T06:07:47.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:07:47.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-06-03T06:07:47.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-06-03T06:07:47.635 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836498 2024-06-03T06:07:47.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:07:47.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836498 2024-06-03T06:07:47.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:07:48.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:07:48.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:07:49.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-06-03T06:07:49.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:07:49.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-06-03T06:07:49.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:07:49.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:07:49.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-06-03T06:07:49.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:07:49.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-06-03T06:07:49.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-06-03T06:07:49.264 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672975 2024-06-03T06:07:49.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:07:49.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672975 2024-06-03T06:07:49.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:07:49.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-06-03T06:07:49.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:07:49.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:07:49.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-06-03T06:07:49.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:07:49.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-06-03T06:07:49.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-06-03T06:07:49.556 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509453 2024-06-03T06:07:49.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:07:49.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-06-03T06:07:49.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:07:49.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345931 2024-06-03T06:07:49.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:07:49.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:07:49.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345931 2024-06-03T06:07:49.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:07:49.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345931 2024-06-03T06:07:49.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-06-03T06:07:49.849 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345931 2024-06-03T06:07:49.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:07:50.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345932 -lt 85899345931 2024-06-03T06:07:50.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:07:50.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182409 2024-06-03T06:07:50.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:07:50.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:07:50.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182409 2024-06-03T06:07:50.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:07:50.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182409 2024-06-03T06:07:50.141 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182409 2024-06-03T06:07:50.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2024-06-03T06:07:50.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:07:50.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182409 -lt 107374182409 2024-06-03T06:07:50.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:07:50.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018887 2024-06-03T06:07:50.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:07:50.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:07:50.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018887 2024-06-03T06:07:50.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:07:50.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-06-03T06:07:50.433 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018887 2024-06-03T06:07:50.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-06-03T06:07:50.434 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:07:50.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018887 2024-06-03T06:07:50.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:07:50.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855364 2024-06-03T06:07:50.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:07:50.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:07:50.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855364 2024-06-03T06:07:50.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:07:50.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855364 2024-06-03T06:07:50.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855364' 2024-06-03T06:07:50.728 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.6 seq 150323855364 2024-06-03T06:07:50.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:07:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855364 2024-06-03T06:07:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:07:51.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691842 2024-06-03T06:07:51.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:07:51.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:07:51.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691842 2024-06-03T06:07:51.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:07:51.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-06-03T06:07:51.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691842' 2024-06-03T06:07:51.062 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.7 seq 171798691842 2024-06-03T06:07:51.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:07:51.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691842 2024-06-03T06:07:51.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:07:51.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:07:51.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:07:51.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-06-03T06:07:51.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:07:51.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:07:51.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:07:51.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:07:51.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:07:51.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:07:51.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:07:52.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-06-03T06:07:52.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:07:52.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:07:52.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:07:52.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-06-03T06:07:52.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:07:52.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:07:52.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2024-06-03T06:07:52.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2024-06-03T06:07:52.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2024-06-03T06:07:52.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2024-06-03T06:07:52.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.44967 bs=1M count=50 2024-06-03T06:07:52.805 INFO:tasks.workunit.client.0.smithi039.stderr:50+0 records in 2024-06-03T06:07:52.805 INFO:tasks.workunit.client.0.smithi039.stderr:50+0 records out 2024-06-03T06:07:52.805 INFO:tasks.workunit.client.0.smithi039.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.141425 s, 371 MB/s 2024-06-03T06:07:52.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2024-06-03T06:07:52.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:07:52.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.44967 2024-06-03T06:07:54.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:07:54.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.44967 2024-06-03T06:07:55.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:07:55.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.44967 2024-06-03T06:07:57.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:07:57.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.44967 2024-06-03T06:07:58.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:07:58.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.44967 2024-06-03T06:08:00.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:00.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.44967 2024-06-03T06:08:01.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:01.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.44967 2024-06-03T06:08:02.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:02.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.44967 2024-06-03T06:08:04.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:04.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.44967 2024-06-03T06:08:05.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:05.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.44967 2024-06-03T06:08:07.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:07.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.44967 2024-06-03T06:08:08.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:08.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.44967 2024-06-03T06:08:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.44967 2024-06-03T06:08:11.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:11.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.44967 2024-06-03T06:08:12.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:12.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.44967 2024-06-03T06:08:14.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:14.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.44967 2024-06-03T06:08:15.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:15.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.44967 2024-06-03T06:08:17.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:17.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.44967 2024-06-03T06:08:18.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:18.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.44967 2024-06-03T06:08:19.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:19.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.44967 2024-06-03T06:08:21.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:21.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.44967 2024-06-03T06:08:22.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:22.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.44967 2024-06-03T06:08:24.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:24.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.44967 2024-06-03T06:08:25.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:25.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.44967 2024-06-03T06:08:27.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:27.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.44967 2024-06-03T06:08:29.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:29.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.44967 2024-06-03T06:08:30.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:30.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.44967 2024-06-03T06:08:32.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:32.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.44967 2024-06-03T06:08:33.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:33.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.44967 2024-06-03T06:08:34.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:34.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.44967 2024-06-03T06:08:36.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:36.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.44967 2024-06-03T06:08:37.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:37.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.44967 2024-06-03T06:08:39.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:39.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.44967 2024-06-03T06:08:40.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:40.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.44967 2024-06-03T06:08:42.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:42.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.44967 2024-06-03T06:08:43.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:43.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.44967 2024-06-03T06:08:45.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:45.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.44967 2024-06-03T06:08:46.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:46.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.44967 2024-06-03T06:08:48.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:48.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.44967 2024-06-03T06:08:49.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-03T06:08:49.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.44967 2024-06-03T06:08:51.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.44967 2024-06-03T06:08:51.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2024-06-03T06:08:51.525 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 3 2024-06-03T06:08:51.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2024-06-03T06:08:51.815 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:08:51.815 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-06-03T06:07:47.076997+0000 49'26 49:54 [5] 5 [5] 5 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.815 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:46.286560+0000 49'13 49:43 [3] 3 [3] 3 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.815 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:46.985093+0000 49'13 49:29 [1] 1 [1] 1 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.815 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:46.984382+0000 49'13 49:32 [1] 1 [1] 1 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.816 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:47.076606+0000 49'13 49:32 [1] 1 [1] 1 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.816 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:07:46.984672+0000 0'0 49:19 [1] 1 [1] 1 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.816 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-06-03T06:07:47.076620+0000 49'26 49:45 [5] 5 [5] 5 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.816 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:46.572456+0000 49'13 49:34 [6] 6 [6] 6 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.816 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:07:46.572430+0000 0'0 49:22 [6] 6 [6] 6 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.816 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-06-03T06:07:45.726011+0000 49'39 49:62 [4] 4 [4] 4 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.816 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-06-03T06:07:46.177572+0000 49'52 49:76 [3] 3 [3] 3 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.816 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-06-03T06:07:45.726012+0000 49'52 49:75 [4] 4 [4] 4 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.816 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:45.555618+0000 49'13 49:40 [0] 0 [0] 0 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.816 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:07:46.286561+0000 0'0 49:22 [3] 3 [3] 3 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.816 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:46.959303+0000 49'13 49:41 [5] 5 [5] 5 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.816 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:45.555772+0000 49'13 49:33 [0] 0 [0] 0 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.817 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:45.664860+0000 49'13 49:43 [7] 7 [7] 7 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.817 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:45.664890+0000 49'13 49:34 [7] 7 [7] 7 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.817 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:46.984990+0000 49'13 49:31 [1] 1 [1] 1 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.817 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:46.572439+0000 49'13 49:32 [6] 6 [6] 6 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.817 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:47.076811+0000 49'13 49:31 [5] 5 [5] 5 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.817 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:45.555918+0000 49'13 49:35 [0] 0 [0] 0 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.817 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-06-03T06:07:45.412626+0000 49'26 49:47 [0] 0 [0] 0 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.817 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:45.857930+0000 49'13 49:32 [4] 4 [4] 4 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.817 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:07:46.572437+0000 0'0 49:18 [2] 2 [2] 2 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.817 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-06-03T06:07:46.984734+0000 49'39 49:56 [1] 1 [1] 1 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.817 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:07:46.698193+0000 0'0 49:17 [6] 6 [6] 6 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.817 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:45.555578+0000 49'13 49:34 [0] 0 [0] 0 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.818 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:07:47.076751+0000 0'0 49:17 [5] 5 [5] 5 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.818 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:46.177591+0000 49'13 49:33 [3] 3 [3] 3 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.818 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:07:46.984828+0000 0'0 49:17 [1] 1 [1] 1 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.818 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-03T06:07:45.556018+0000 49'13 49:34 [0] 0 [0] 0 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:51.818 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:08:51.818 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:08:51.818 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:08:51.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2024-06-03T06:08:51.963 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T06:08:51.964 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_recovery_max_active": "0" 2024-06-03T06:08:51.964 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:08:51.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2024-06-03T06:08:52.072 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T06:08:52.072 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_recovery_max_active_hdd": "3" 2024-06-03T06:08:52.072 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:08:52.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2024-06-03T06:08:52.231 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T06:08:52.231 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_recovery_max_active_ssd": "10" 2024-06-03T06:08:52.231 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:08:52.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2024-06-03T06:08:52.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-06-03T06:08:52.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-06-03T06:08:52.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-06-03T06:08:52.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-06-03T06:08:52.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-06-03T06:08:52.725 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:08:52.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-06-03T06:08:52.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-06-03T06:08:54.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2024-06-03T06:08:54.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2024-06-03T06:08:54.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2024-06-03T06:08:54.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-06-03T06:08:54.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-06-03T06:08:54.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-06-03T06:08:54.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-06-03T06:08:54.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-06-03T06:08:55.026 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:08:55.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-06-03T06:08:55.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-06-03T06:08:57.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2024-06-03T06:08:57.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2024-06-03T06:08:57.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2024-06-03T06:08:57.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-06-03T06:08:57.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-06-03T06:08:57.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-06-03T06:08:57.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-06-03T06:08:57.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-06-03T06:08:57.329 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:08:57.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-06-03T06:08:57.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-06-03T06:08:59.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 2 -eq 10 2024-06-03T06:08:59.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 2 + 1 2024-06-03T06:08:59.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=3 2024-06-03T06:08:59.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-06-03T06:08:59.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-06-03T06:08:59.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-06-03T06:08:59.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-06-03T06:08:59.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-06-03T06:08:59.634 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:08:59.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 2 -ge 2 2024-06-03T06:08:59.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:288: TEST_recovery_scrub_2: break 2024-06-03T06:08:59.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:298: TEST_recovery_scrub_2: ceph pg dump pgs 2024-06-03T06:08:59.921 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:08:59.921 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-06-03T06:08:53.435465+0000 49'26 52:63 [5,0,2] 5 [5,0,2] 5 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.921 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.394752+0000 49'13 52:53 [3,0,7] 3 [3,0,7] 3 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.921 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-06-03T06:08:53.118250+0000 49'26 52:53 [1,3,0] 1 [1,3,0] 1 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.921 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.118175+0000 49'13 52:42 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.921 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:52.948139+0000 49'13 52:42 [1,6,2] 1 [1,6,2] 1 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.921 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:08:53.118542+0000 0'0 52:31 [1,4,5] 1 [1,4,5] 1 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:59.921 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-06-03T06:08:53.436013+0000 49'26 52:53 [5,4,3] 5 [5,4,3] 5 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.921 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-06-03T06:08:53.275670+0000 49'13 52:45 [6,7,3] 6 [6,7,3] 6 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.274276+0000 49'13 52:45 [6,7,3] 6 [6,7,3] 6 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-06-03T06:08:53.578761+0000 49'39 52:71 [4,2,6] 4 [4,2,6] 4 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 4 0 8 0 0 209715200 0 0 52 0 52 active+recovering+degraded 2024-06-03T06:08:53.275483+0000 49'52 52:86 [3,0,6] 3 [3,0,6] 3 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-06-03T06:08:53.579170+0000 49'52 52:84 [4,1,2] 4 [4,1,2] 4 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.864688+0000 49'13 52:49 [0,7,1] 0 [0,7,1] 0 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:08:53.233749+0000 0'0 52:33 [3,0,4] 3 [3,0,4] 3 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:59.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.545436+0000 49'13 52:50 [5,0,4] 5 [5,0,4] 5 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.578770+0000 49'13 52:43 [0,3,4] 0 [0,3,4] 0 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.545429+0000 49'13 52:53 [7,0,2] 7 [7,0,2] 7 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.436328+0000 49'13 52:43 [7,4,2] 7 [7,4,2] 7 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.118411+0000 49'13 52:41 [1,4,2] 1 [1,4,2] 1 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.688141+0000 49'13 52:41 [6,0,3] 6 [6,0,3] 6 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.393900+0000 49'13 52:39 [5,7,6] 5 [5,7,6] 5 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.689025+0000 49'13 52:44 [0,1,3] 0 [0,1,3] 0 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-06-03T06:08:53.688582+0000 49'26 52:56 [0,1,6] 0 [0,1,6] 0 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.435604+0000 49'13 52:40 [4,6,3] 4 [4,6,3] 4 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 peering 2024-06-03T06:08:51.795416+0000 0'0 50:20 [2,0,4] 2 [2,0,4] 2 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-06-03T06:08:53.234096+0000 49'39 52:66 [1,5,3] 1 [1,5,3] 1 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:08:53.234158+0000 0'0 52:28 [6,0,7] 6 [6,0,7] 6 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:59.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.764979+0000 49'13 52:43 [0,4,5] 0 [0,4,5] 0 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:08:53.273780+0000 0'0 52:27 [5,2,3] 5 [5,2,3] 5 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:59.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.394924+0000 49'13 52:42 [3,0,4] 3 [3,0,4] 3 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-03T06:08:53.117974+0000 0'0 52:29 [1,7,4] 1 [1,7,4] 1 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 periodic scrub scheduled @ 2024-06-04T06:07:44.619806+0000 0 0 2024-06-03T06:08:59.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-03T06:08:53.545612+0000 49'13 52:43 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-03T06:07:44.619806+0000 0'0 2024-06-03T06:07:44.619806+0000 0 0 no scrub is scheduled 0 0 2024-06-03T06:08:59.924 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:08:59.924 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:08:59.924 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:08:59.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pids= 2024-06-03T06:08:59.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: recov_scrub_count=0 2024-06-03T06:08:59.932 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: expr 32 - 1 2024-06-03T06:08:59.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: seq 0 31 2024-06-03T06:08:59.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 0 2024-06-03T06:08:59.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2024-06-03T06:08:59.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66463"' 2024-06-03T06:08:59.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66463' 2024-06-03T06:08:59.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.0 2024-06-03T06:08:59.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 1 2024-06-03T06:08:59.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66466: /' 2024-06-03T06:08:59.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2024-06-03T06:08:59.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66468"' 2024-06-03T06:08:59.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66468' 2024-06-03T06:08:59.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1 2024-06-03T06:08:59.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 2 2024-06-03T06:08:59.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66473: /' 2024-06-03T06:08:59.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2024-06-03T06:08:59.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66476"' 2024-06-03T06:08:59.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66476' 2024-06-03T06:08:59.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.2 2024-06-03T06:08:59.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66480: /' 2024-06-03T06:08:59.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 3 2024-06-03T06:08:59.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2024-06-03T06:08:59.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66482"' 2024-06-03T06:08:59.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66482' 2024-06-03T06:08:59.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.3 2024-06-03T06:08:59.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 4 2024-06-03T06:08:59.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2024-06-03T06:08:59.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66489"' 2024-06-03T06:08:59.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66489' 2024-06-03T06:08:59.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66486: /' 2024-06-03T06:08:59.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 5 2024-06-03T06:08:59.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2024-06-03T06:08:59.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66493"' 2024-06-03T06:08:59.943 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66493' 2024-06-03T06:08:59.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.943 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 6 2024-06-03T06:08:59.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66497: /' 2024-06-03T06:08:59.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.5 2024-06-03T06:08:59.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2024-06-03T06:08:59.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66500"' 2024-06-03T06:08:59.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66500' 2024-06-03T06:08:59.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 7 2024-06-03T06:08:59.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.6 2024-06-03T06:08:59.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2024-06-03T06:08:59.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66506"' 2024-06-03T06:08:59.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66506' 2024-06-03T06:08:59.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66505: /' 2024-06-03T06:08:59.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.7 2024-06-03T06:08:59.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.4 2024-06-03T06:08:59.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66498: /' 2024-06-03T06:08:59.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 8 2024-06-03T06:08:59.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2024-06-03T06:08:59.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66515"' 2024-06-03T06:08:59.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66515' 2024-06-03T06:08:59.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66518: /' 2024-06-03T06:08:59.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 9 2024-06-03T06:08:59.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2024-06-03T06:08:59.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66519"' 2024-06-03T06:08:59.953 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66519' 2024-06-03T06:08:59.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.8 2024-06-03T06:08:59.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66523: /' 2024-06-03T06:08:59.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.9 2024-06-03T06:08:59.954 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 10 2024-06-03T06:08:59.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2024-06-03T06:08:59.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66527"' 2024-06-03T06:08:59.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66527' 2024-06-03T06:08:59.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 11 2024-06-03T06:08:59.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2024-06-03T06:08:59.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66529"' 2024-06-03T06:08:59.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66529' 2024-06-03T06:08:59.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 12 2024-06-03T06:08:59.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2024-06-03T06:08:59.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66531"' 2024-06-03T06:08:59.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66531' 2024-06-03T06:08:59.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66534: /' 2024-06-03T06:08:59.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 13 2024-06-03T06:08:59.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2024-06-03T06:08:59.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66539"' 2024-06-03T06:08:59.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66539' 2024-06-03T06:08:59.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 14 2024-06-03T06:08:59.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66510: /' 2024-06-03T06:08:59.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2024-06-03T06:08:59.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66543"' 2024-06-03T06:08:59.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66543' 2024-06-03T06:08:59.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.e 2024-06-03T06:08:59.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66538: /' 2024-06-03T06:08:59.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 15 2024-06-03T06:08:59.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2024-06-03T06:08:59.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66554"' 2024-06-03T06:08:59.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66554' 2024-06-03T06:08:59.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 16 2024-06-03T06:08:59.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2024-06-03T06:08:59.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66556"' 2024-06-03T06:08:59.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66556' 2024-06-03T06:08:59.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.10 2024-06-03T06:08:59.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.f 2024-06-03T06:08:59.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66559: /' 2024-06-03T06:08:59.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.c 2024-06-03T06:08:59.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66551: /' 2024-06-03T06:08:59.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 17 2024-06-03T06:08:59.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66546: /' 2024-06-03T06:08:59.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2024-06-03T06:08:59.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66567"' 2024-06-03T06:08:59.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66567' 2024-06-03T06:08:59.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.11 2024-06-03T06:08:59.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.a 2024-06-03T06:08:59.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66562: /' 2024-06-03T06:08:59.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.b 2024-06-03T06:08:59.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 18 2024-06-03T06:08:59.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2024-06-03T06:08:59.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:08:59.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:08:59.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66576"' 2024-06-03T06:08:59.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66576' 2024-06-03T06:08:59.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:08:59.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66579: /' 2024-06-03T06:08:59.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66570: /' 2024-06-03T06:08:59.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 19 2024-06-03T06:08:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.d 2024-06-03T06:09:00.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2024-06-03T06:09:00.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:09:00.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:09:00.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66589"' 2024-06-03T06:09:00.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66589' 2024-06-03T06:09:00.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:09:00.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66581: /' 2024-06-03T06:09:00.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66596: /' 2024-06-03T06:09:00.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 20 2024-06-03T06:09:00.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.13 2024-06-03T06:09:00.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2024-06-03T06:09:00.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:09:00.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:09:00.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66598"' 2024-06-03T06:09:00.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66598' 2024-06-03T06:09:00.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:09:00.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.12 2024-06-03T06:09:00.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 21 2024-06-03T06:09:00.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2024-06-03T06:09:00.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:09:00.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:09:00.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66603"' 2024-06-03T06:09:00.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66603' 2024-06-03T06:09:00.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:09:00.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66602: /' 2024-06-03T06:09:00.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 22 2024-06-03T06:09:00.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2024-06-03T06:09:00.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:09:00.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:09:00.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66607"' 2024-06-03T06:09:00.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66607' 2024-06-03T06:09:00.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:09:00.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 23 2024-06-03T06:09:00.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2024-06-03T06:09:00.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:09:00.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:09:00.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66613"' 2024-06-03T06:09:00.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66613' 2024-06-03T06:09:00.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:09:00.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.14 2024-06-03T06:09:00.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 24 2024-06-03T06:09:00.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2024-06-03T06:09:00.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:09:00.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:09:00.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66617"' 2024-06-03T06:09:00.068 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66617' 2024-06-03T06:09:00.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:09:00.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.15 2024-06-03T06:09:00.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66610: /' 2024-06-03T06:09:00.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66622: /' 2024-06-03T06:09:00.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.16 2024-06-03T06:09:00.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 25 2024-06-03T06:09:00.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.17 2024-06-03T06:09:00.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2024-06-03T06:09:00.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:09:00.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:09:00.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66631"' 2024-06-03T06:09:00.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66631' 2024-06-03T06:09:00.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:09:00.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66625: /' 2024-06-03T06:09:00.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.18 2024-06-03T06:09:00.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66629: /' 2024-06-03T06:09:00.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 26 2024-06-03T06:09:00.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2024-06-03T06:09:00.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:09:00.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:09:00.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66649"' 2024-06-03T06:09:00.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66649' 2024-06-03T06:09:00.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:09:00.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.19 2024-06-03T06:09:00.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66641: /' 2024-06-03T06:09:00.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 27 2024-06-03T06:09:00.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1a 2024-06-03T06:09:00.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2024-06-03T06:09:00.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:09:00.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:09:00.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66661"' 2024-06-03T06:09:00.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66661' 2024-06-03T06:09:00.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:09:00.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66654: /' 2024-06-03T06:09:00.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 28 2024-06-03T06:09:00.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2024-06-03T06:09:00.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:09:00.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:09:00.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66672"' 2024-06-03T06:09:00.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66672' 2024-06-03T06:09:00.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:09:00.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1b 2024-06-03T06:09:00.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66664: /' 2024-06-03T06:09:00.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 29 2024-06-03T06:09:00.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2024-06-03T06:09:00.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:09:00.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:09:00.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66677"' 2024-06-03T06:09:00.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66677' 2024-06-03T06:09:00.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:09:00.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1c 2024-06-03T06:09:00.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66676: /' 2024-06-03T06:09:00.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 30 2024-06-03T06:09:00.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2024-06-03T06:09:00.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:09:00.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:09:00.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66686"' 2024-06-03T06:09:00.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66686' 2024-06-03T06:09:00.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-03T06:09:00.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1d 2024-06-03T06:09:00.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 31 2024-06-03T06:09:00.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2024-06-03T06:09:00.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:09:00.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:09:00.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66696"' 2024-06-03T06:09:00.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66696' 2024-06-03T06:09:00.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:306: TEST_recovery_scrub_2: wait_background_check pids 2024-06-03T06:09:00.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: pids=' 66463 66468 66476 66482 66489 66493 66500 66506 66515 66519 66527 66529 66531 66539 66543 66554 66556 66567 66576 66589 66598 66603 66607 66613 66617 66631 66649 66661 66672 66677 66686 66696' 2024-06-03T06:09:00.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: return_code=0 2024-06-03T06:09:00.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:00.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66463 2024-06-03T06:09:00.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1e 2024-06-03T06:09:00.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66689: /' 2024-06-03T06:09:00.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66694: /' 2024-06-03T06:09:00.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66701: /' 2024-06-03T06:09:00.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1f 2024-06-03T06:09:05.955 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.4 2024-06-03T06:09:05.955 INFO:tasks.workunit.client.0.smithi039.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.4 2024-06-03T06:09:05.955 INFO:tasks.workunit.client.0.smithi039.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-06-03T06:09:05.955 INFO:tasks.workunit.client.0.smithi039.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:05.955 INFO:tasks.workunit.client.0.smithi039.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-06-03T06:09:05.955 INFO:tasks.workunit.client.0.smithi039.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.4 query 2024-06-03T06:09:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.4 2024-06-03T06:09:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:66497: instructing pg 1.4 on osd.3 to scrub 2024-06-03T06:09:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2024-06-03T06:09:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:66497: dumped pgs 2024-06-03T06:09:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-06-03T06:09:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.4 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.4 2024-06-03T06:09:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-06-03T06:09:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-06-03T06:09:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-06-03T06:09:07.585 INFO:tasks.workunit.client.0.smithi039.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1b 2024-06-03T06:09:07.585 INFO:tasks.workunit.client.0.smithi039.stderr:66664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1b 2024-06-03T06:09:07.585 INFO:tasks.workunit.client.0.smithi039.stderr:66664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-06-03T06:09:07.585 INFO:tasks.workunit.client.0.smithi039.stderr:66664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:07.586 INFO:tasks.workunit.client.0.smithi039.stderr:66664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:07.586 INFO:tasks.workunit.client.0.smithi039.stderr:66664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-06-03T06:09:07.586 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:07.586 INFO:tasks.workunit.client.0.smithi039.stderr:66664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1b query 2024-06-03T06:09:07.586 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:07.586 INFO:tasks.workunit.client.0.smithi039.stderr:66664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:07.586 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:07.586 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:07.586 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1b 2024-06-03T06:09:07.586 INFO:tasks.workunit.client.0.smithi039.stderr:66664: instructing pg 1.1b on osd.0 to scrub 2024-06-03T06:09:07.586 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:07.586 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:07.587 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2024-06-03T06:09:07.587 INFO:tasks.workunit.client.0.smithi039.stderr:66664: dumped pgs 2024-06-03T06:09:07.587 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1b 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:07.587 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1b 2024-06-03T06:09:07.587 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:07.587 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:07.587 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:07.587 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:07.587 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:07.587 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:07.587 INFO:tasks.workunit.client.0.smithi039.stderr:66664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2024-06-03T06:09:07.587 INFO:tasks.workunit.client.0.smithi039.stderr:66664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-06-03T06:09:07.587 INFO:tasks.workunit.client.0.smithi039.stderr:66664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:07.587 INFO:tasks.workunit.client.0.smithi039.stderr:66664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:07.588 INFO:tasks.workunit.client.0.smithi039.stderr:66664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-06-03T06:09:07.588 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:07.588 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:07.588 INFO:tasks.workunit.client.0.smithi039.stderr:66664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:07.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:07.597 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.8 2024-06-03T06:09:07.597 INFO:tasks.workunit.client.0.smithi039.stderr:66518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.8 2024-06-03T06:09:07.597 INFO:tasks.workunit.client.0.smithi039.stderr:66518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-06-03T06:09:07.597 INFO:tasks.workunit.client.0.smithi039.stderr:66518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:07.597 INFO:tasks.workunit.client.0.smithi039.stderr:66518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-06-03T06:09:07.597 INFO:tasks.workunit.client.0.smithi039.stderr:66518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:07.597 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:07.597 INFO:tasks.workunit.client.0.smithi039.stderr:66518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.8 query 2024-06-03T06:09:07.597 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:07.598 INFO:tasks.workunit.client.0.smithi039.stderr:66518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:07.598 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:07.598 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:07.598 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.8 2024-06-03T06:09:07.598 INFO:tasks.workunit.client.0.smithi039.stderr:66518: instructing pg 1.8 on osd.5 to scrub 2024-06-03T06:09:07.598 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:07.598 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2024-06-03T06:09:07.598 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:07.598 INFO:tasks.workunit.client.0.smithi039.stderr:66518: dumped pgs 2024-06-03T06:09:07.598 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.8 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:07.598 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.8 2024-06-03T06:09:07.598 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:07.598 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:07.599 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:07.599 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:07.599 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:07.599 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:07.599 INFO:tasks.workunit.client.0.smithi039.stderr:66518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2024-06-03T06:09:07.599 INFO:tasks.workunit.client.0.smithi039.stderr:66518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-06-03T06:09:07.599 INFO:tasks.workunit.client.0.smithi039.stderr:66518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:07.599 INFO:tasks.workunit.client.0.smithi039.stderr:66518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-06-03T06:09:07.599 INFO:tasks.workunit.client.0.smithi039.stderr:66518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:07.599 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:07.599 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:07.599 INFO:tasks.workunit.client.0.smithi039.stderr:66518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:07.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:08.075 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1 2024-06-03T06:09:08.075 INFO:tasks.workunit.client.0.smithi039.stderr:66473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1 2024-06-03T06:09:08.075 INFO:tasks.workunit.client.0.smithi039.stderr:66473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-06-03T06:09:08.075 INFO:tasks.workunit.client.0.smithi039.stderr:66473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:08.075 INFO:tasks.workunit.client.0.smithi039.stderr:66473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:08.076 INFO:tasks.workunit.client.0.smithi039.stderr:66473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-06-03T06:09:08.076 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.076 INFO:tasks.workunit.client.0.smithi039.stderr:66473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:08.076 INFO:tasks.workunit.client.0.smithi039.stderr:66473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1 query 2024-06-03T06:09:08.076 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:08.076 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:08.076 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:08.076 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1 2024-06-03T06:09:08.076 INFO:tasks.workunit.client.0.smithi039.stderr:66473: instructing pg 1.1 on osd.5 to scrub 2024-06-03T06:09:08.076 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:08.076 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:08.076 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2024-06-03T06:09:08.076 INFO:tasks.workunit.client.0.smithi039.stderr:66473: dumped pgs 2024-06-03T06:09:08.076 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1 2024-06-03T06:09:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:66473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-06-03T06:09:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:66473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-06-03T06:09:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:66473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:66473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-06-03T06:09:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:66473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:08.078 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:08.078 INFO:tasks.workunit.client.0.smithi039.stderr:66473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:08.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:08.165 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.7 2024-06-03T06:09:08.165 INFO:tasks.workunit.client.0.smithi039.stderr:66510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.7 2024-06-03T06:09:08.165 INFO:tasks.workunit.client.0.smithi039.stderr:66510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-06-03T06:09:08.165 INFO:tasks.workunit.client.0.smithi039.stderr:66510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:08.165 INFO:tasks.workunit.client.0.smithi039.stderr:66510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:08.166 INFO:tasks.workunit.client.0.smithi039.stderr:66510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-06-03T06:09:08.166 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.166 INFO:tasks.workunit.client.0.smithi039.stderr:66510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:08.166 INFO:tasks.workunit.client.0.smithi039.stderr:66510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.7 query 2024-06-03T06:09:08.166 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:08.166 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:08.166 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:08.166 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.7 2024-06-03T06:09:08.166 INFO:tasks.workunit.client.0.smithi039.stderr:66510: instructing pg 1.7 on osd.6 to scrub 2024-06-03T06:09:08.166 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:08.166 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2024-06-03T06:09:08.166 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:08.166 INFO:tasks.workunit.client.0.smithi039.stderr:66510: dumped pgs 2024-06-03T06:09:08.167 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-06-03T06:09:08.167 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.7 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.167 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.7 2024-06-03T06:09:08.167 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.167 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:08.167 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:08.167 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:08.167 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:08.167 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:08.167 INFO:tasks.workunit.client.0.smithi039.stderr:66510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-06-03T06:09:08.167 INFO:tasks.workunit.client.0.smithi039.stderr:66510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-06-03T06:09:08.167 INFO:tasks.workunit.client.0.smithi039.stderr:66510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:08.167 INFO:tasks.workunit.client.0.smithi039.stderr:66510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-06-03T06:09:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:66510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-06-03T06:09:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-06-03T06:09:08.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-06-03T06:09:08.412 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1c 2024-06-03T06:09:08.412 INFO:tasks.workunit.client.0.smithi039.stderr:66676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1c 2024-06-03T06:09:08.412 INFO:tasks.workunit.client.0.smithi039.stderr:66676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-06-03T06:09:08.412 INFO:tasks.workunit.client.0.smithi039.stderr:66676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:08.412 INFO:tasks.workunit.client.0.smithi039.stderr:66676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-06-03T06:09:08.412 INFO:tasks.workunit.client.0.smithi039.stderr:66676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:08.412 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.412 INFO:tasks.workunit.client.0.smithi039.stderr:66676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1c query 2024-06-03T06:09:08.413 INFO:tasks.workunit.client.0.smithi039.stderr:66676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:08.413 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:08.413 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:08.413 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:08.413 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1c 2024-06-03T06:09:08.413 INFO:tasks.workunit.client.0.smithi039.stderr:66676: instructing pg 1.1c on osd.0 to scrub 2024-06-03T06:09:08.413 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2024-06-03T06:09:08.413 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:08.413 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:08.413 INFO:tasks.workunit.client.0.smithi039.stderr:66676: dumped pgs 2024-06-03T06:09:08.413 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1c 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.413 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1c 2024-06-03T06:09:08.413 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.414 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:08.414 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:08.414 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:08.414 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:08.414 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:08.414 INFO:tasks.workunit.client.0.smithi039.stderr:66676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2024-06-03T06:09:08.414 INFO:tasks.workunit.client.0.smithi039.stderr:66676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-06-03T06:09:08.414 INFO:tasks.workunit.client.0.smithi039.stderr:66676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:08.414 INFO:tasks.workunit.client.0.smithi039.stderr:66676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:08.414 INFO:tasks.workunit.client.0.smithi039.stderr:66676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-06-03T06:09:08.414 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:08.414 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:08.414 INFO:tasks.workunit.client.0.smithi039.stderr:66676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:08.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:08.417 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.d 2024-06-03T06:09:08.417 INFO:tasks.workunit.client.0.smithi039.stderr:66579: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.d 2024-06-03T06:09:08.418 INFO:tasks.workunit.client.0.smithi039.stderr:66579: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-06-03T06:09:08.418 INFO:tasks.workunit.client.0.smithi039.stderr:66579: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:08.418 INFO:tasks.workunit.client.0.smithi039.stderr:66579: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:08.418 INFO:tasks.workunit.client.0.smithi039.stderr:66579: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-06-03T06:09:08.418 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.418 INFO:tasks.workunit.client.0.smithi039.stderr:66579: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.d query 2024-06-03T06:09:08.418 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:08.418 INFO:tasks.workunit.client.0.smithi039.stderr:66579: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:08.418 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:08.418 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:08.418 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.d 2024-06-03T06:09:08.418 INFO:tasks.workunit.client.0.smithi039.stderr:66579: instructing pg 1.d on osd.7 to scrub 2024-06-03T06:09:08.418 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2024-06-03T06:09:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:66579: dumped pgs 2024-06-03T06:09:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.d 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.d 2024-06-03T06:09:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:66579: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2024-06-03T06:09:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:66579: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-06-03T06:09:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:66579: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:08.420 INFO:tasks.workunit.client.0.smithi039.stderr:66579: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:08.420 INFO:tasks.workunit.client.0.smithi039.stderr:66579: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-06-03T06:09:08.420 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:08.420 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:08.420 INFO:tasks.workunit.client.0.smithi039.stderr:66579: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:08.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:08.477 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.5 2024-06-03T06:09:08.477 INFO:tasks.workunit.client.0.smithi039.stderr:66498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.5 2024-06-03T06:09:08.477 INFO:tasks.workunit.client.0.smithi039.stderr:66498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-06-03T06:09:08.478 INFO:tasks.workunit.client.0.smithi039.stderr:66498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:08.478 INFO:tasks.workunit.client.0.smithi039.stderr:66498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:08.478 INFO:tasks.workunit.client.0.smithi039.stderr:66498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-06-03T06:09:08.478 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.478 INFO:tasks.workunit.client.0.smithi039.stderr:66498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.5 query 2024-06-03T06:09:08.478 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:08.478 INFO:tasks.workunit.client.0.smithi039.stderr:66498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:08.478 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:08.478 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:08.478 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.5 2024-06-03T06:09:08.478 INFO:tasks.workunit.client.0.smithi039.stderr:66498: instructing pg 1.5 on osd.4 to scrub 2024-06-03T06:09:08.478 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:08.479 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:08.479 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2024-06-03T06:09:08.479 INFO:tasks.workunit.client.0.smithi039.stderr:66498: dumped pgs 2024-06-03T06:09:08.479 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.5 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.479 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.5 2024-06-03T06:09:08.479 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.479 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:08.479 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:08.479 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:08.479 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:08.479 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:08.479 INFO:tasks.workunit.client.0.smithi039.stderr:66498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-06-03T06:09:08.479 INFO:tasks.workunit.client.0.smithi039.stderr:66498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-06-03T06:09:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:66498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:66498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:66498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-06-03T06:09:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:66498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:08.635 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.2 2024-06-03T06:09:08.635 INFO:tasks.workunit.client.0.smithi039.stderr:66480: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.2 2024-06-03T06:09:08.635 INFO:tasks.workunit.client.0.smithi039.stderr:66480: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-06-03T06:09:08.635 INFO:tasks.workunit.client.0.smithi039.stderr:66480: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:08.635 INFO:tasks.workunit.client.0.smithi039.stderr:66480: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-06-03T06:09:08.635 INFO:tasks.workunit.client.0.smithi039.stderr:66480: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:08.636 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.636 INFO:tasks.workunit.client.0.smithi039.stderr:66480: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.2 query 2024-06-03T06:09:08.636 INFO:tasks.workunit.client.0.smithi039.stderr:66480: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:08.636 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:08.636 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:08.636 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:08.636 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.2 2024-06-03T06:09:08.636 INFO:tasks.workunit.client.0.smithi039.stderr:66480: instructing pg 1.2 on osd.0 to scrub 2024-06-03T06:09:08.636 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:08.636 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:08.637 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2024-06-03T06:09:08.637 INFO:tasks.workunit.client.0.smithi039.stderr:66480: dumped pgs 2024-06-03T06:09:08.637 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.2 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.637 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.2 2024-06-03T06:09:08.637 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.637 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:08.637 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:08.637 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:08.637 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:08.637 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:08.637 INFO:tasks.workunit.client.0.smithi039.stderr:66480: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-06-03T06:09:08.637 INFO:tasks.workunit.client.0.smithi039.stderr:66480: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-06-03T06:09:08.637 INFO:tasks.workunit.client.0.smithi039.stderr:66480: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:08.638 INFO:tasks.workunit.client.0.smithi039.stderr:66480: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:08.638 INFO:tasks.workunit.client.0.smithi039.stderr:66480: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-06-03T06:09:08.638 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:08.638 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:08.638 INFO:tasks.workunit.client.0.smithi039.stderr:66480: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:08.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:08.949 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.18 2024-06-03T06:09:08.950 INFO:tasks.workunit.client.0.smithi039.stderr:66629: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.18 2024-06-03T06:09:08.950 INFO:tasks.workunit.client.0.smithi039.stderr:66629: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-06-03T06:09:08.950 INFO:tasks.workunit.client.0.smithi039.stderr:66629: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:08.950 INFO:tasks.workunit.client.0.smithi039.stderr:66629: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-06-03T06:09:08.950 INFO:tasks.workunit.client.0.smithi039.stderr:66629: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:08.950 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.950 INFO:tasks.workunit.client.0.smithi039.stderr:66629: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.18 query 2024-06-03T06:09:08.950 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:08.950 INFO:tasks.workunit.client.0.smithi039.stderr:66629: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:08.950 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:08.950 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:08.951 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.18 2024-06-03T06:09:08.951 INFO:tasks.workunit.client.0.smithi039.stderr:66629: instructing pg 1.18 on osd.5 to scrub 2024-06-03T06:09:08.951 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:08.951 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:08.951 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2024-06-03T06:09:08.951 INFO:tasks.workunit.client.0.smithi039.stderr:66629: dumped pgs 2024-06-03T06:09:08.951 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.18 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.951 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.18 2024-06-03T06:09:08.951 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.951 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:08.951 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:08.951 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:08.951 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:08.951 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:08.952 INFO:tasks.workunit.client.0.smithi039.stderr:66629: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2024-06-03T06:09:08.952 INFO:tasks.workunit.client.0.smithi039.stderr:66629: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-06-03T06:09:08.952 INFO:tasks.workunit.client.0.smithi039.stderr:66629: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:08.952 INFO:tasks.workunit.client.0.smithi039.stderr:66629: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:08.952 INFO:tasks.workunit.client.0.smithi039.stderr:66629: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-06-03T06:09:08.952 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:09:03.972008+0000 '>' '' 2024-06-03T06:09:08.952 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:08.952 INFO:tasks.workunit.client.0.smithi039.stderr:66629: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:08.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:08.997 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1d 2024-06-03T06:09:08.998 INFO:tasks.workunit.client.0.smithi039.stderr:66689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1d 2024-06-03T06:09:08.998 INFO:tasks.workunit.client.0.smithi039.stderr:66689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-06-03T06:09:08.998 INFO:tasks.workunit.client.0.smithi039.stderr:66689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:08.998 INFO:tasks.workunit.client.0.smithi039.stderr:66689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-06-03T06:09:08.998 INFO:tasks.workunit.client.0.smithi039.stderr:66689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:08.998 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.998 INFO:tasks.workunit.client.0.smithi039.stderr:66689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1d query 2024-06-03T06:09:08.998 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:08.998 INFO:tasks.workunit.client.0.smithi039.stderr:66689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:08.998 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:08.998 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:08.999 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1d 2024-06-03T06:09:08.999 INFO:tasks.workunit.client.0.smithi039.stderr:66689: instructing pg 1.1d on osd.1 to scrub 2024-06-03T06:09:08.999 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2024-06-03T06:09:08.999 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:08.999 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:08.999 INFO:tasks.workunit.client.0.smithi039.stderr:66689: dumped pgs 2024-06-03T06:09:08.999 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1d 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.999 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1d 2024-06-03T06:09:08.999 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:08.999 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:08.999 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:08.999 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:08.999 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:08.999 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.000 INFO:tasks.workunit.client.0.smithi039.stderr:66689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2024-06-03T06:09:09.000 INFO:tasks.workunit.client.0.smithi039.stderr:66689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-06-03T06:09:09.000 INFO:tasks.workunit.client.0.smithi039.stderr:66689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.000 INFO:tasks.workunit.client.0.smithi039.stderr:66689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-06-03T06:09:09.000 INFO:tasks.workunit.client.0.smithi039.stderr:66689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.000 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.000 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.000 INFO:tasks.workunit.client.0.smithi039.stderr:66689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.054 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.3 2024-06-03T06:09:09.054 INFO:tasks.workunit.client.0.smithi039.stderr:66486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.3 2024-06-03T06:09:09.054 INFO:tasks.workunit.client.0.smithi039.stderr:66486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:09:09.054 INFO:tasks.workunit.client.0.smithi039.stderr:66486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.054 INFO:tasks.workunit.client.0.smithi039.stderr:66486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:09:09.054 INFO:tasks.workunit.client.0.smithi039.stderr:66486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.055 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.055 INFO:tasks.workunit.client.0.smithi039.stderr:66486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.055 INFO:tasks.workunit.client.0.smithi039.stderr:66486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.3 query 2024-06-03T06:09:09.055 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.055 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.055 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.055 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.3 2024-06-03T06:09:09.055 INFO:tasks.workunit.client.0.smithi039.stderr:66486: instructing pg 1.3 on osd.4 to scrub 2024-06-03T06:09:09.055 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2024-06-03T06:09:09.055 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.055 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.055 INFO:tasks.workunit.client.0.smithi039.stderr:66486: dumped pgs 2024-06-03T06:09:09.055 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.3 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.3 2024-06-03T06:09:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:66486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:09:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:66486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:09:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:66486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:66486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:09:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:66486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.057 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.057 INFO:tasks.workunit.client.0.smithi039.stderr:66486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.127 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.16 2024-06-03T06:09:09.127 INFO:tasks.workunit.client.0.smithi039.stderr:66625: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.16 2024-06-03T06:09:09.127 INFO:tasks.workunit.client.0.smithi039.stderr:66625: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-06-03T06:09:09.127 INFO:tasks.workunit.client.0.smithi039.stderr:66625: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:66625: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-06-03T06:09:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:66625: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:66625: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:66625: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.16 query 2024-06-03T06:09:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.16 2024-06-03T06:09:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:66625: instructing pg 1.16 on osd.6 to scrub 2024-06-03T06:09:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2024-06-03T06:09:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:66625: dumped pgs 2024-06-03T06:09:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.16 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.16 2024-06-03T06:09:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:66625: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2024-06-03T06:09:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:66625: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-06-03T06:09:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:66625: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:66625: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-06-03T06:09:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:66625: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.130 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:09:06.401215+0000 '>' '' 2024-06-03T06:09:09.130 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.130 INFO:tasks.workunit.client.0.smithi039.stderr:66625: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.264 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.a 2024-06-03T06:09:09.264 INFO:tasks.workunit.client.0.smithi039.stderr:66551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.a 2024-06-03T06:09:09.264 INFO:tasks.workunit.client.0.smithi039.stderr:66551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-06-03T06:09:09.264 INFO:tasks.workunit.client.0.smithi039.stderr:66551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.264 INFO:tasks.workunit.client.0.smithi039.stderr:66551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-06-03T06:09:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:66551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:66551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.a query 2024-06-03T06:09:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:66551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.a 2024-06-03T06:09:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:66551: instructing pg 1.a on osd.1 to scrub 2024-06-03T06:09:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2024-06-03T06:09:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:66551: dumped pgs 2024-06-03T06:09:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.a 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.a 2024-06-03T06:09:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:66551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2024-06-03T06:09:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:66551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-06-03T06:09:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:66551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:66551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-06-03T06:09:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:66551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.267 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.267 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.267 INFO:tasks.workunit.client.0.smithi039.stderr:66551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.298 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.14 2024-06-03T06:09:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:66602: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.14 2024-06-03T06:09:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:66602: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-06-03T06:09:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:66602: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:66602: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:66602: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-06-03T06:09:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:66602: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.14 query 2024-06-03T06:09:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:66602: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.14 2024-06-03T06:09:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:66602: instructing pg 1.14 on osd.2 to scrub 2024-06-03T06:09:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2024-06-03T06:09:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:66602: dumped pgs 2024-06-03T06:09:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.14 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.14 2024-06-03T06:09:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:66602: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2024-06-03T06:09:09.301 INFO:tasks.workunit.client.0.smithi039.stderr:66602: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-06-03T06:09:09.301 INFO:tasks.workunit.client.0.smithi039.stderr:66602: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.301 INFO:tasks.workunit.client.0.smithi039.stderr:66602: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-06-03T06:09:09.301 INFO:tasks.workunit.client.0.smithi039.stderr:66602: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.301 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.301 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.301 INFO:tasks.workunit.client.0.smithi039.stderr:66602: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.427 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.e 2024-06-03T06:09:09.427 INFO:tasks.workunit.client.0.smithi039.stderr:66546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.e 2024-06-03T06:09:09.427 INFO:tasks.workunit.client.0.smithi039.stderr:66546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-06-03T06:09:09.427 INFO:tasks.workunit.client.0.smithi039.stderr:66546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.427 INFO:tasks.workunit.client.0.smithi039.stderr:66546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.428 INFO:tasks.workunit.client.0.smithi039.stderr:66546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-06-03T06:09:09.428 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.428 INFO:tasks.workunit.client.0.smithi039.stderr:66546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.428 INFO:tasks.workunit.client.0.smithi039.stderr:66546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.e query 2024-06-03T06:09:09.428 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.428 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.428 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.428 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.e 2024-06-03T06:09:09.428 INFO:tasks.workunit.client.0.smithi039.stderr:66546: instructing pg 1.e on osd.1 to scrub 2024-06-03T06:09:09.428 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.428 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.428 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2024-06-03T06:09:09.428 INFO:tasks.workunit.client.0.smithi039.stderr:66546: dumped pgs 2024-06-03T06:09:09.428 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.e 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.e 2024-06-03T06:09:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:66546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2024-06-03T06:09:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:66546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-06-03T06:09:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:66546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:66546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:66546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-06-03T06:09:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:66546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.430 INFO:tasks.workunit.client.0.smithi039.stderr:_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.430 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.430 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.430 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-06-03T06:09:09.430 INFO:tasks.workunit.client.0.smithi039.stderr:66510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-06-03T06:09:09.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-06-03T06:09:09.532 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.19 2024-06-03T06:09:09.532 INFO:tasks.workunit.client.0.smithi039.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.19 2024-06-03T06:09:09.532 INFO:tasks.workunit.client.0.smithi039.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-06-03T06:09:09.532 INFO:tasks.workunit.client.0.smithi039.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.532 INFO:tasks.workunit.client.0.smithi039.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-06-03T06:09:09.532 INFO:tasks.workunit.client.0.smithi039.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.532 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.532 INFO:tasks.workunit.client.0.smithi039.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.19 query 2024-06-03T06:09:09.533 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.533 INFO:tasks.workunit.client.0.smithi039.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.533 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.533 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.533 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.19 2024-06-03T06:09:09.533 INFO:tasks.workunit.client.0.smithi039.stderr:66641: instructing pg 1.19 on osd.3 to scrub 2024-06-03T06:09:09.533 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2024-06-03T06:09:09.533 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.533 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.533 INFO:tasks.workunit.client.0.smithi039.stderr:66641: dumped pgs 2024-06-03T06:09:09.533 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.19 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.533 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.19 2024-06-03T06:09:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2024-06-03T06:09:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-06-03T06:09:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-06-03T06:09:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.535 INFO:tasks.workunit.client.0.smithi039.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.603 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.17 2024-06-03T06:09:09.603 INFO:tasks.workunit.client.0.smithi039.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.17 2024-06-03T06:09:09.603 INFO:tasks.workunit.client.0.smithi039.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-06-03T06:09:09.604 INFO:tasks.workunit.client.0.smithi039.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.604 INFO:tasks.workunit.client.0.smithi039.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-06-03T06:09:09.604 INFO:tasks.workunit.client.0.smithi039.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.604 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.604 INFO:tasks.workunit.client.0.smithi039.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.604 INFO:tasks.workunit.client.0.smithi039.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.17 query 2024-06-03T06:09:09.604 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.604 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.604 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.604 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.17 2024-06-03T06:09:09.604 INFO:tasks.workunit.client.0.smithi039.stderr:66622: instructing pg 1.17 on osd.0 to scrub 2024-06-03T06:09:09.604 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2024-06-03T06:09:09.604 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.605 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.605 INFO:tasks.workunit.client.0.smithi039.stderr:66622: dumped pgs 2024-06-03T06:09:09.605 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.17 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.605 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.17 2024-06-03T06:09:09.605 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.605 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.605 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.605 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.605 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.606 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.606 INFO:tasks.workunit.client.0.smithi039.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2024-06-03T06:09:09.606 INFO:tasks.workunit.client.0.smithi039.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-06-03T06:09:09.606 INFO:tasks.workunit.client.0.smithi039.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.606 INFO:tasks.workunit.client.0.smithi039.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.606 INFO:tasks.workunit.client.0.smithi039.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-06-03T06:09:09.606 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.606 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.606 INFO:tasks.workunit.client.0.smithi039.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.623 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1e 2024-06-03T06:09:09.623 INFO:tasks.workunit.client.0.smithi039.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1e 2024-06-03T06:09:09.623 INFO:tasks.workunit.client.0.smithi039.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-06-03T06:09:09.623 INFO:tasks.workunit.client.0.smithi039.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.623 INFO:tasks.workunit.client.0.smithi039.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.623 INFO:tasks.workunit.client.0.smithi039.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-06-03T06:09:09.624 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.624 INFO:tasks.workunit.client.0.smithi039.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1e query 2024-06-03T06:09:09.624 INFO:tasks.workunit.client.0.smithi039.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.624 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.624 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.624 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.624 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1e 2024-06-03T06:09:09.624 INFO:tasks.workunit.client.0.smithi039.stderr:66694: instructing pg 1.1e on osd.3 to scrub 2024-06-03T06:09:09.624 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2024-06-03T06:09:09.625 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.625 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.625 INFO:tasks.workunit.client.0.smithi039.stderr:66694: dumped pgs 2024-06-03T06:09:09.625 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1e 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.625 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1e 2024-06-03T06:09:09.625 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.625 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.625 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.625 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.626 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.626 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.626 INFO:tasks.workunit.client.0.smithi039.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2024-06-03T06:09:09.626 INFO:tasks.workunit.client.0.smithi039.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-06-03T06:09:09.626 INFO:tasks.workunit.client.0.smithi039.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.626 INFO:tasks.workunit.client.0.smithi039.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-06-03T06:09:09.626 INFO:tasks.workunit.client.0.smithi039.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.627 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.627 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.627 INFO:tasks.workunit.client.0.smithi039.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.742 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.9 2024-06-03T06:09:09.742 INFO:tasks.workunit.client.0.smithi039.stderr:66523: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.9 2024-06-03T06:09:09.743 INFO:tasks.workunit.client.0.smithi039.stderr:66523: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-06-03T06:09:09.743 INFO:tasks.workunit.client.0.smithi039.stderr:66523: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.743 INFO:tasks.workunit.client.0.smithi039.stderr:66523: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.743 INFO:tasks.workunit.client.0.smithi039.stderr:66523: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-06-03T06:09:09.743 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.743 INFO:tasks.workunit.client.0.smithi039.stderr:66523: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.9 query 2024-06-03T06:09:09.743 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.743 INFO:tasks.workunit.client.0.smithi039.stderr:66523: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.743 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.743 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.743 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.9 2024-06-03T06:09:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:66523: instructing pg 1.9 on osd.1 to scrub 2024-06-03T06:09:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2024-06-03T06:09:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:66523: dumped pgs 2024-06-03T06:09:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.9 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.9 2024-06-03T06:09:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:66523: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2024-06-03T06:09:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:66523: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-06-03T06:09:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:66523: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:66523: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:66523: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-06-03T06:09:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:09:06.336032+0000 '>' '' 2024-06-03T06:09:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:66523: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.748 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1a 2024-06-03T06:09:09.748 INFO:tasks.workunit.client.0.smithi039.stderr:66654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1a 2024-06-03T06:09:09.749 INFO:tasks.workunit.client.0.smithi039.stderr:66654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-06-03T06:09:09.749 INFO:tasks.workunit.client.0.smithi039.stderr:66654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.749 INFO:tasks.workunit.client.0.smithi039.stderr:66654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-06-03T06:09:09.749 INFO:tasks.workunit.client.0.smithi039.stderr:66654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.749 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.749 INFO:tasks.workunit.client.0.smithi039.stderr:66654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.749 INFO:tasks.workunit.client.0.smithi039.stderr:66654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1a query 2024-06-03T06:09:09.749 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.749 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.749 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.749 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1a 2024-06-03T06:09:09.749 INFO:tasks.workunit.client.0.smithi039.stderr:66654: instructing pg 1.1a on osd.1 to scrub 2024-06-03T06:09:09.749 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.749 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.750 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2024-06-03T06:09:09.750 INFO:tasks.workunit.client.0.smithi039.stderr:66654: dumped pgs 2024-06-03T06:09:09.750 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1a 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.750 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1a 2024-06-03T06:09:09.750 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.750 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.750 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.750 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.750 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.750 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.750 INFO:tasks.workunit.client.0.smithi039.stderr:66654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2024-06-03T06:09:09.750 INFO:tasks.workunit.client.0.smithi039.stderr:66654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-06-03T06:09:09.751 INFO:tasks.workunit.client.0.smithi039.stderr:66654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.751 INFO:tasks.workunit.client.0.smithi039.stderr:66654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.751 INFO:tasks.workunit.client.0.smithi039.stderr:66654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-06-03T06:09:09.751 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:09:05.336501+0000 '>' '' 2024-06-03T06:09:09.751 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.751 INFO:tasks.workunit.client.0.smithi039.stderr:66654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.756 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.10 2024-06-03T06:09:09.756 INFO:tasks.workunit.client.0.smithi039.stderr:66559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.10 2024-06-03T06:09:09.756 INFO:tasks.workunit.client.0.smithi039.stderr:66559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-06-03T06:09:09.756 INFO:tasks.workunit.client.0.smithi039.stderr:66559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.756 INFO:tasks.workunit.client.0.smithi039.stderr:66559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-06-03T06:09:09.756 INFO:tasks.workunit.client.0.smithi039.stderr:66559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.756 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.756 INFO:tasks.workunit.client.0.smithi039.stderr:66559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.756 INFO:tasks.workunit.client.0.smithi039.stderr:66559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.10 query 2024-06-03T06:09:09.756 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.757 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.757 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.757 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.10 2024-06-03T06:09:09.757 INFO:tasks.workunit.client.0.smithi039.stderr:66559: instructing pg 1.10 on osd.5 to scrub 2024-06-03T06:09:09.757 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.757 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2024-06-03T06:09:09.757 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.757 INFO:tasks.workunit.client.0.smithi039.stderr:66559: dumped pgs 2024-06-03T06:09:09.757 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.10 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.757 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.10 2024-06-03T06:09:09.757 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.758 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.758 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.758 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.758 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.758 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.758 INFO:tasks.workunit.client.0.smithi039.stderr:66559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2024-06-03T06:09:09.758 INFO:tasks.workunit.client.0.smithi039.stderr:66559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-06-03T06:09:09.758 INFO:tasks.workunit.client.0.smithi039.stderr:66559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.758 INFO:tasks.workunit.client.0.smithi039.stderr:66559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-06-03T06:09:09.758 INFO:tasks.workunit.client.0.smithi039.stderr:66559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.758 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.759 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.759 INFO:tasks.workunit.client.0.smithi039.stderr:66559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.867 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.13 2024-06-03T06:09:09.867 INFO:tasks.workunit.client.0.smithi039.stderr:66596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.13 2024-06-03T06:09:09.867 INFO:tasks.workunit.client.0.smithi039.stderr:66596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-06-03T06:09:09.867 INFO:tasks.workunit.client.0.smithi039.stderr:66596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.867 INFO:tasks.workunit.client.0.smithi039.stderr:66596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-06-03T06:09:09.867 INFO:tasks.workunit.client.0.smithi039.stderr:66596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.867 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.868 INFO:tasks.workunit.client.0.smithi039.stderr:66596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.868 INFO:tasks.workunit.client.0.smithi039.stderr:66596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.13 query 2024-06-03T06:09:09.868 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.868 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.868 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.868 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.13 2024-06-03T06:09:09.868 INFO:tasks.workunit.client.0.smithi039.stderr:66596: instructing pg 1.13 on osd.4 to scrub 2024-06-03T06:09:09.868 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2024-06-03T06:09:09.868 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.868 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.868 INFO:tasks.workunit.client.0.smithi039.stderr:66596: dumped pgs 2024-06-03T06:09:09.868 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.13 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.868 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.13 2024-06-03T06:09:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:66596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2024-06-03T06:09:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:66596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-06-03T06:09:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:66596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:66596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:66596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-06-03T06:09:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:66596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.926 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.11 2024-06-03T06:09:09.926 INFO:tasks.workunit.client.0.smithi039.stderr:66570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.11 2024-06-03T06:09:09.926 INFO:tasks.workunit.client.0.smithi039.stderr:66570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-06-03T06:09:09.926 INFO:tasks.workunit.client.0.smithi039.stderr:66570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.926 INFO:tasks.workunit.client.0.smithi039.stderr:66570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-06-03T06:09:09.926 INFO:tasks.workunit.client.0.smithi039.stderr:66570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.926 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.926 INFO:tasks.workunit.client.0.smithi039.stderr:66570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:66570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.11 query 2024-06-03T06:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.11 2024-06-03T06:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:66570: instructing pg 1.11 on osd.0 to scrub 2024-06-03T06:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2024-06-03T06:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:66570: dumped pgs 2024-06-03T06:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.11 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.11 2024-06-03T06:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.928 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.928 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.928 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.928 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.928 INFO:tasks.workunit.client.0.smithi039.stderr:66570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2024-06-03T06:09:09.928 INFO:tasks.workunit.client.0.smithi039.stderr:66570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-06-03T06:09:09.928 INFO:tasks.workunit.client.0.smithi039.stderr:66570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.928 INFO:tasks.workunit.client.0.smithi039.stderr:66570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-06-03T06:09:09.928 INFO:tasks.workunit.client.0.smithi039.stderr:66570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.928 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.928 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.928 INFO:tasks.workunit.client.0.smithi039.stderr:66570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.951 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.b 2024-06-03T06:09:09.951 INFO:tasks.workunit.client.0.smithi039.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.b 2024-06-03T06:09:09.951 INFO:tasks.workunit.client.0.smithi039.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-06-03T06:09:09.951 INFO:tasks.workunit.client.0.smithi039.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.951 INFO:tasks.workunit.client.0.smithi039.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.951 INFO:tasks.workunit.client.0.smithi039.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-06-03T06:09:09.951 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.951 INFO:tasks.workunit.client.0.smithi039.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.951 INFO:tasks.workunit.client.0.smithi039.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.b query 2024-06-03T06:09:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.b 2024-06-03T06:09:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:66538: instructing pg 1.b on osd.1 to scrub 2024-06-03T06:09:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2024-06-03T06:09:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:66538: dumped pgs 2024-06-03T06:09:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.b 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.b 2024-06-03T06:09:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2024-06-03T06:09:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-06-03T06:09:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-06-03T06:09:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.954 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.954 INFO:tasks.workunit.client.0.smithi039.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.961 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1f 2024-06-03T06:09:09.961 INFO:tasks.workunit.client.0.smithi039.stderr:66701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1f 2024-06-03T06:09:09.961 INFO:tasks.workunit.client.0.smithi039.stderr:66701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-06-03T06:09:09.961 INFO:tasks.workunit.client.0.smithi039.stderr:66701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.961 INFO:tasks.workunit.client.0.smithi039.stderr:66701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.962 INFO:tasks.workunit.client.0.smithi039.stderr:66701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-06-03T06:09:09.962 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.962 INFO:tasks.workunit.client.0.smithi039.stderr:66701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1f query 2024-06-03T06:09:09.962 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.962 INFO:tasks.workunit.client.0.smithi039.stderr:66701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.962 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.962 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.962 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1f 2024-06-03T06:09:09.962 INFO:tasks.workunit.client.0.smithi039.stderr:66701: instructing pg 1.1f on osd.5 to scrub 2024-06-03T06:09:09.962 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2024-06-03T06:09:09.963 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.963 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.963 INFO:tasks.workunit.client.0.smithi039.stderr:66701: dumped pgs 2024-06-03T06:09:09.963 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1f 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.963 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1f 2024-06-03T06:09:09.963 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.963 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.963 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.963 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.963 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.963 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.963 INFO:tasks.workunit.client.0.smithi039.stderr:66701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2024-06-03T06:09:09.964 INFO:tasks.workunit.client.0.smithi039.stderr:66701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-06-03T06:09:09.964 INFO:tasks.workunit.client.0.smithi039.stderr:66701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.964 INFO:tasks.workunit.client.0.smithi039.stderr:66701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.964 INFO:tasks.workunit.client.0.smithi039.stderr:66701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-06-03T06:09:09.964 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.964 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.964 INFO:tasks.workunit.client.0.smithi039.stderr:66701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.974 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.0 2024-06-03T06:09:09.974 INFO:tasks.workunit.client.0.smithi039.stderr:66466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.0 2024-06-03T06:09:09.974 INFO:tasks.workunit.client.0.smithi039.stderr:66466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:09:09.974 INFO:tasks.workunit.client.0.smithi039.stderr:66466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.974 INFO:tasks.workunit.client.0.smithi039.stderr:66466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.974 INFO:tasks.workunit.client.0.smithi039.stderr:66466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:09:09.974 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.974 INFO:tasks.workunit.client.0.smithi039.stderr:66466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.0 query 2024-06-03T06:09:09.975 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.975 INFO:tasks.workunit.client.0.smithi039.stderr:66466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.975 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.975 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.975 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.0 2024-06-03T06:09:09.975 INFO:tasks.workunit.client.0.smithi039.stderr:66466: instructing pg 1.0 on osd.7 to scrub 2024-06-03T06:09:09.975 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.975 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2024-06-03T06:09:09.975 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.975 INFO:tasks.workunit.client.0.smithi039.stderr:66466: dumped pgs 2024-06-03T06:09:09.975 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.0 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.975 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.0 2024-06-03T06:09:09.975 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:66466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:09:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:66466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:09:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:66466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:66466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:66466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:09:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:66466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:09.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:09.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:09.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:09.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66468 2024-06-03T06:09:09.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:09.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66476 2024-06-03T06:09:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66482 2024-06-03T06:09:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66489 2024-06-03T06:09:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-06-03T06:09:09.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-06-03T06:09:09.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 0 + 1 2024-06-03T06:09:09.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=1 2024-06-03T06:09:09.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:09.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66493 2024-06-03T06:09:09.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:09.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:09.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:09.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:09.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66500 2024-06-03T06:09:09.982 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.f 2024-06-03T06:09:09.982 INFO:tasks.workunit.client.0.smithi039.stderr:66562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.f 2024-06-03T06:09:09.982 INFO:tasks.workunit.client.0.smithi039.stderr:66562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.f query 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.f 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: instructing pg 1.f on osd.6 to scrub 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: dumped pgs 2024-06-03T06:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.f 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.f 2024-06-03T06:09:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:66562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2024-06-03T06:09:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:66562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-06-03T06:09:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:66562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:66562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:66562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-06-03T06:09:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:66562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:09.985 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.15 2024-06-03T06:09:09.985 INFO:tasks.workunit.client.0.smithi039.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.15 2024-06-03T06:09:09.985 INFO:tasks.workunit.client.0.smithi039.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-06-03T06:09:09.985 INFO:tasks.workunit.client.0.smithi039.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.985 INFO:tasks.workunit.client.0.smithi039.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.985 INFO:tasks.workunit.client.0.smithi039.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-06-03T06:09:09.985 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.985 INFO:tasks.workunit.client.0.smithi039.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:09.985 INFO:tasks.workunit.client.0.smithi039.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.15 query 2024-06-03T06:09:09.985 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:09.985 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:09.985 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:09.985 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.15 2024-06-03T06:09:09.986 INFO:tasks.workunit.client.0.smithi039.stderr:66610: instructing pg 1.15 on osd.1 to scrub 2024-06-03T06:09:09.986 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2024-06-03T06:09:09.986 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:09.986 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:09.986 INFO:tasks.workunit.client.0.smithi039.stderr:66610: dumped pgs 2024-06-03T06:09:09.986 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.15 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.986 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.15 2024-06-03T06:09:09.986 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:09.986 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:09.986 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:09.986 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:09.986 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:09.986 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:09.987 INFO:tasks.workunit.client.0.smithi039.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2024-06-03T06:09:09.987 INFO:tasks.workunit.client.0.smithi039.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-06-03T06:09:09.987 INFO:tasks.workunit.client.0.smithi039.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:09.987 INFO:tasks.workunit.client.0.smithi039.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:09.987 INFO:tasks.workunit.client.0.smithi039.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-06-03T06:09:09.987 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:09.987 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:09.987 INFO:tasks.workunit.client.0.smithi039.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:09.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:10.016 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.6 2024-06-03T06:09:10.016 INFO:tasks.workunit.client.0.smithi039.stderr:66505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.6 2024-06-03T06:09:10.016 INFO:tasks.workunit.client.0.smithi039.stderr:66505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-06-03T06:09:10.017 INFO:tasks.workunit.client.0.smithi039.stderr:66505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:10.017 INFO:tasks.workunit.client.0.smithi039.stderr:66505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:10.017 INFO:tasks.workunit.client.0.smithi039.stderr:66505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-06-03T06:09:10.017 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:10.017 INFO:tasks.workunit.client.0.smithi039.stderr:66505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.6 query 2024-06-03T06:09:10.017 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:10.017 INFO:tasks.workunit.client.0.smithi039.stderr:66505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:10.017 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:10.017 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:10.017 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.6 2024-06-03T06:09:10.017 INFO:tasks.workunit.client.0.smithi039.stderr:66505: instructing pg 1.6 on osd.6 to scrub 2024-06-03T06:09:10.018 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2024-06-03T06:09:10.018 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:10.018 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:10.018 INFO:tasks.workunit.client.0.smithi039.stderr:66505: dumped pgs 2024-06-03T06:09:10.018 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.6 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:10.018 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.6 2024-06-03T06:09:10.018 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:10.018 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:10.018 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:10.018 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:10.018 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:10.018 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:10.018 INFO:tasks.workunit.client.0.smithi039.stderr:66505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-06-03T06:09:10.019 INFO:tasks.workunit.client.0.smithi039.stderr:66505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-06-03T06:09:10.019 INFO:tasks.workunit.client.0.smithi039.stderr:66505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:10.019 INFO:tasks.workunit.client.0.smithi039.stderr:66505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:10.019 INFO:tasks.workunit.client.0.smithi039.stderr:66505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-06-03T06:09:10.019 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:10.019 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:10.019 INFO:tasks.workunit.client.0.smithi039.stderr:66505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:10.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:10.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66506 2024-06-03T06:09:10.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-06-03T06:09:10.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-06-03T06:09:10.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 1 + 1 2024-06-03T06:09:10.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=2 2024-06-03T06:09:10.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66515 2024-06-03T06:09:10.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66519 2024-06-03T06:09:10.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66527 2024-06-03T06:09:10.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66529 2024-06-03T06:09:10.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66531 2024-06-03T06:09:10.024 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.12 2024-06-03T06:09:10.024 INFO:tasks.workunit.client.0.smithi039.stderr:66581: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.12 2024-06-03T06:09:10.025 INFO:tasks.workunit.client.0.smithi039.stderr:66581: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-06-03T06:09:10.025 INFO:tasks.workunit.client.0.smithi039.stderr:66581: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:10.025 INFO:tasks.workunit.client.0.smithi039.stderr:66581: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-06-03T06:09:10.025 INFO:tasks.workunit.client.0.smithi039.stderr:66581: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:10.025 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:10.025 INFO:tasks.workunit.client.0.smithi039.stderr:66581: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.12 query 2024-06-03T06:09:10.025 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:10.025 INFO:tasks.workunit.client.0.smithi039.stderr:66581: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:10.025 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:10.025 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:10.025 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.12 2024-06-03T06:09:10.026 INFO:tasks.workunit.client.0.smithi039.stderr:66581: instructing pg 1.12 on osd.0 to scrub 2024-06-03T06:09:10.026 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2024-06-03T06:09:10.026 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:10.026 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:10.026 INFO:tasks.workunit.client.0.smithi039.stderr:66581: dumped pgs 2024-06-03T06:09:10.026 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.12 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:10.026 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.12 2024-06-03T06:09:10.026 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:10.026 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:10.026 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:10.026 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:10.026 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:10.027 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:10.027 INFO:tasks.workunit.client.0.smithi039.stderr:66581: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2024-06-03T06:09:10.027 INFO:tasks.workunit.client.0.smithi039.stderr:66581: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-06-03T06:09:10.027 INFO:tasks.workunit.client.0.smithi039.stderr:66581: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:10.027 INFO:tasks.workunit.client.0.smithi039.stderr:66581: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-06-03T06:09:10.027 INFO:tasks.workunit.client.0.smithi039.stderr:66581: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:10.027 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:07:44.619806+0000 '>' '' 2024-06-03T06:09:10.027 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:10.027 INFO:tasks.workunit.client.0.smithi039.stderr:66581: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:10.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:10.103 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.c 2024-06-03T06:09:10.103 INFO:tasks.workunit.client.0.smithi039.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.c 2024-06-03T06:09:10.103 INFO:tasks.workunit.client.0.smithi039.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-06-03T06:09:10.103 INFO:tasks.workunit.client.0.smithi039.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:10.103 INFO:tasks.workunit.client.0.smithi039.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:10.104 INFO:tasks.workunit.client.0.smithi039.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-06-03T06:09:10.104 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:10.104 INFO:tasks.workunit.client.0.smithi039.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.c query 2024-06-03T06:09:10.104 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-03T06:09:10.104 INFO:tasks.workunit.client.0.smithi039.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-03T06:09:10.104 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-03T06:09:10.104 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-03T06:09:10.104 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.c 2024-06-03T06:09:10.104 INFO:tasks.workunit.client.0.smithi039.stderr:66534: instructing pg 1.c on osd.3 to scrub 2024-06-03T06:09:10.104 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-03T06:09:10.104 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2024-06-03T06:09:10.104 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-03T06:09:10.104 INFO:tasks.workunit.client.0.smithi039.stderr:66534: dumped pgs 2024-06-03T06:09:10.104 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.c 2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.c 2024-06-03T06:09:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-03T06:07:44.619806+0000 2024-06-03T06:09:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-03T06:09:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-03T06:09:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-03T06:09:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-03T06:09:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-03T06:09:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2024-06-03T06:09:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-06-03T06:09:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:09:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-06-03T06:09:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:09:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-03T06:09:06.684338+0000 '>' '' 2024-06-03T06:09:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-03T06:09:10.106 INFO:tasks.workunit.client.0.smithi039.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-03T06:09:10.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:09:10.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66539 2024-06-03T06:09:10.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66543 2024-06-03T06:09:10.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66554 2024-06-03T06:09:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66556 2024-06-03T06:09:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66567 2024-06-03T06:09:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66576 2024-06-03T06:09:10.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66589 2024-06-03T06:09:10.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66598 2024-06-03T06:09:10.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66603 2024-06-03T06:09:10.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66607 2024-06-03T06:09:10.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66613 2024-06-03T06:09:10.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66617 2024-06-03T06:09:10.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66631 2024-06-03T06:09:10.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66649 2024-06-03T06:09:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66661 2024-06-03T06:09:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66672 2024-06-03T06:09:10.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66677 2024-06-03T06:09:10.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66686 2024-06-03T06:09:10.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-03T06:09:10.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66696 2024-06-03T06:09:10.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-03T06:09:10.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-03T06:09:10.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-03T06:09:10.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: eval 'pids='\'''\''' 2024-06-03T06:09:10.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: pids= 2024-06-03T06:09:10.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:229: wait_background_check: return 0 2024-06-03T06:09:10.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:307: TEST_recovery_scrub_2: return_code=0 2024-06-03T06:09:10.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:308: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2024-06-03T06:09:10.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: ERRORS=0 2024-06-03T06:09:10.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:311: TEST_recovery_scrub_2: test 2 -eq 0 2024-06-03T06:09:10.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2024-06-03T06:09:10.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2024-06-03T06:09:10.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/osd.5.pid 2024-06-03T06:09:10.114 INFO:tasks.workunit.client.0.smithi039.stderr:td/osd-recovery-scrub/mon.a.pid 2024-06-03T06:09:10.114 INFO:tasks.workunit.client.0.smithi039.stderr:td/osd-recovery-scrub/mgr.x.pid 2024-06-03T06:09:10.114 INFO:tasks.workunit.client.0.smithi039.stderr:td/osd-recovery-scrub/osd.4.pid 2024-06-03T06:09:10.114 INFO:tasks.workunit.client.0.smithi039.stderr:td/osd-recovery-scrub/osd.0.pid 2024-06-03T06:09:10.114 INFO:tasks.workunit.client.0.smithi039.stderr:td/osd-recovery-scrub/osd.3.pid 2024-06-03T06:09:10.114 INFO:tasks.workunit.client.0.smithi039.stderr:td/osd-recovery-scrub/osd.6.pid 2024-06-03T06:09:10.114 INFO:tasks.workunit.client.0.smithi039.stderr:td/osd-recovery-scrub/osd.2.pid 2024-06-03T06:09:10.114 INFO:tasks.workunit.client.0.smithi039.stderr:td/osd-recovery-scrub/osd.1.pid 2024-06-03T06:09:10.114 INFO:tasks.workunit.client.0.smithi039.stderr:td/osd-recovery-scrub/osd.7.pid' 2024-06-03T06:09:10.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/osd.6.pid td/osd-recovery-scrub/osd.2.pid td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/osd.7.pid 2024-06-03T06:09:10.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: pid='61496 2024-06-03T06:09:10.114 INFO:tasks.workunit.client.0.smithi039.stderr:53585 2024-06-03T06:09:10.115 INFO:tasks.workunit.client.0.smithi039.stderr:53669 2024-06-03T06:09:10.115 INFO:tasks.workunit.client.0.smithi039.stderr:60120 2024-06-03T06:09:10.115 INFO:tasks.workunit.client.0.smithi039.stderr:54616 2024-06-03T06:09:10.115 INFO:tasks.workunit.client.0.smithi039.stderr:58744 2024-06-03T06:09:10.115 INFO:tasks.workunit.client.0.smithi039.stderr:62872 2024-06-03T06:09:10.115 INFO:tasks.workunit.client.0.smithi039.stderr:57368 2024-06-03T06:09:10.115 INFO:tasks.workunit.client.0.smithi039.stderr:55992 2024-06-03T06:09:10.115 INFO:tasks.workunit.client.0.smithi039.stderr:64248' 2024-06-03T06:09:10.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: kill -0 61496 53585 53669 60120 54616 58744 62872 57368 55992 64248 2024-06-03T06:09:10.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:327: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2024-06-03T06:09:10.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:09:10.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:09:10.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:09:10.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:09:10.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:09:15.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:09:15.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: declare -a err_strings 2024-06-03T06:09:15.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:330: TEST_recovery_scrub_2: err_strings[0]='not scheduling scrubs due to active recovery' 2024-06-03T06:09:15.415 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: expr 8 - 1 2024-06-03T06:09:15.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: seq 0 7 2024-06-03T06:09:15.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2024-06-03T06:09:15.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2024-06-03T06:09:15.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2024-06-03T06:09:15.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2024-06-03T06:09:15.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2024-06-03T06:09:15.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2024-06-03T06:09:15.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2024-06-03T06:09:15.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2024-06-03T06:09:15.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:336: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2024-06-03T06:09:15.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:338: TEST_recovery_scrub_2: found=false 2024-06-03T06:09:15.449 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: expr 8 - 1 2024-06-03T06:09:15.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: seq 0 7 2024-06-03T06:09:15.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2024-06-03T06:09:15.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2024-06-03T06:09:15.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2024-06-03T06:09:15.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2024-06-03T06:09:15.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2024-06-03T06:09:15.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2024-06-03T06:09:15.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2024-06-03T06:09:15.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T06:09:15.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2024-06-03T06:09:15.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:346: TEST_recovery_scrub_2: '[' false = true ']' 2024-06-03T06:09:15.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:352: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2024-06-03T06:09:15.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-06-03T06:09:15.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:09:15.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-06-03T06:09:15.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:09:15.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:09:15.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:09:15.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:09:15.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:09:15.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:09:15.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:09:15.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:09:15.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:09:15.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:09:15.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:09:15.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:09:15.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:09:15.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:09:15.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:09:15.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:09:15.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:09:15.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:09:15.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-06-03T06:09:15.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:09:15.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:09:15.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:09:15.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44967 2024-06-03T06:09:15.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:09:15.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:09:15.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:354: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2024-06-03T06:09:15.517 INFO:tasks.workunit.client.0.smithi039.stdout:TEST PASSED 2024-06-03T06:09:15.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:360: TEST_recovery_scrub_2: echo 'TEST PASSED' 2024-06-03T06:09:15.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:361: TEST_recovery_scrub_2: return 0 2024-06-03T06:09:15.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-06-03T06:09:15.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-recovery-scrub 0 2024-06-03T06:09:15.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-06-03T06:09:15.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-03T06:09:15.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-06-03T06:09:15.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:09:15.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:09:15.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:09:15.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:09:15.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:09:15.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:09:15.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:09:15.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:09:15.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:09:15.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:09:15.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:09:15.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:09:15.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:09:15.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:09:15.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:09:15.524 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:09:15.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:09:15.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-03T06:09:15.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-06-03T06:09:15.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:09:15.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:09:15.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44967 2024-06-03T06:09:15.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44967 2024-06-03T06:09:15.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:09:15.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:09:15.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-06-03T06:09:15.529 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-03T06:09:15.529 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-03T06:09:15.596 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-06-03T06:09:15.597 DEBUG:teuthology.orchestra.run.smithi039:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-06-03T06:09:15.660 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-03T06:09:15.664 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-03T06:09:15.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-06-03T06:09:15.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-06-03T06:09:15.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-06-03T06:09:15.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-06-03T06:09:15.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-06-03T06:09:15.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-06-03T06:09:15.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-06-03T06:09:15.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-06-03T06:09:15.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-dump 2024-06-03T06:09:15.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-06-03T06:09:15.665 INFO:tasks.workunit.client.0.smithi039.stdout:This test is disabled 2024-06-03T06:09:15.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-06-03T06:09:15.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-06-03T06:09:15.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-dump 0 2024-06-03T06:09:15.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2024-06-03T06:09:15.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-03T06:09:15.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2024-06-03T06:09:15.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:09:15.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:09:15.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:09:15.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:09:15.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:09:15.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:09:15.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:09:15.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:09:15.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:09:15.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:09:15.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:09:15.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:09:15.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:09:15.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:09:15.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:09:15.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:09:15.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:09:15.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-03T06:09:15.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2024-06-03T06:09:15.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:09:15.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:09:15.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69934 2024-06-03T06:09:15.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69934 2024-06-03T06:09:15.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:09:15.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:09:15.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-06-03T06:09:15.675 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-03T06:09:15.675 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-03T06:09:15.740 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-06-03T06:09:15.741 DEBUG:teuthology.orchestra.run.smithi039:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-06-03T06:09:15.802 INFO:tasks.workunit.client.0.smithi039.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-06-03T06:09:15.802 INFO:tasks.workunit.client.0.smithi039.stderr:++ TIMEOUT=300 2024-06-03T06:09:15.802 INFO:tasks.workunit.client.0.smithi039.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-06-03T06:09:15.802 INFO:tasks.workunit.client.0.smithi039.stderr:++ MAX_TIMEOUT=15 2024-06-03T06:09:15.802 INFO:tasks.workunit.client.0.smithi039.stderr:++ PG_NUM=4 2024-06-03T06:09:15.802 INFO:tasks.workunit.client.0.smithi039.stderr:++ TMPDIR=/tmp 2024-06-03T06:09:15.802 INFO:tasks.workunit.client.0.smithi039.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-06-03T06:09:15.802 INFO:tasks.workunit.client.0.smithi039.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-06-03T06:09:15.803 INFO:tasks.workunit.client.0.smithi039.stderr:++ type xmlstarlet 2024-06-03T06:09:15.803 INFO:tasks.workunit.client.0.smithi039.stderr:++ XMLSTARLET=xmlstarlet 2024-06-03T06:09:15.803 INFO:tasks.workunit.client.0.smithi039.stderr:+++ uname 2024-06-03T06:09:15.803 INFO:tasks.workunit.client.0.smithi039.stderr:++ '[' Linux = FreeBSD ']' 2024-06-03T06:09:15.803 INFO:tasks.workunit.client.0.smithi039.stderr:++ SED=sed 2024-06-03T06:09:15.803 INFO:tasks.workunit.client.0.smithi039.stderr:++ AWK=awk 2024-06-03T06:09:15.804 INFO:tasks.workunit.client.0.smithi039.stderr:+++ stty -a 2024-06-03T06:09:15.804 INFO:tasks.workunit.client.0.smithi039.stderr:+++ head -1 2024-06-03T06:09:15.804 INFO:tasks.workunit.client.0.smithi039.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-06-03T06:09:15.804 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-03T06:09:15.805 INFO:tasks.workunit.client.0.smithi039.stderr:++ termwidth= 2024-06-03T06:09:15.805 INFO:tasks.workunit.client.0.smithi039.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-06-03T06:09:15.805 INFO:tasks.workunit.client.0.smithi039.stderr:++ DIFFCOLOPTS='-y ' 2024-06-03T06:09:15.805 INFO:tasks.workunit.client.0.smithi039.stderr:++ KERNCORE=kernel.core_pattern 2024-06-03T06:09:15.805 INFO:tasks.workunit.client.0.smithi039.stderr:++ EXTRA_OPTS= 2024-06-03T06:09:15.807 INFO:tasks.workunit.client.0.smithi039.stderr:++ test '' = TESTS 2024-06-03T06:09:15.808 INFO:tasks.workunit.client.0.smithi039.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-06-03T06:09:15.808 INFO:tasks.workunit.client.0.smithi039.stderr:++ uname 2024-06-03T06:09:15.809 INFO:tasks.workunit.client.0.smithi039.stderr:+ '[' Linux = FreeBSD ']' 2024-06-03T06:09:15.809 INFO:tasks.workunit.client.0.smithi039.stderr:+ use_ec_overwrite=true 2024-06-03T06:09:15.810 INFO:tasks.workunit.client.0.smithi039.stderr:+ getjson=no 2024-06-03T06:09:15.810 INFO:tasks.workunit.client.0.smithi039.stderr:+ jqfilter='def walk(f): 2024-06-03T06:09:15.810 INFO:tasks.workunit.client.0.smithi039.stderr: . as $in 2024-06-03T06:09:15.810 INFO:tasks.workunit.client.0.smithi039.stderr: | if type == "object" then 2024-06-03T06:09:15.810 INFO:tasks.workunit.client.0.smithi039.stderr: reduce keys[] as $key 2024-06-03T06:09:15.810 INFO:tasks.workunit.client.0.smithi039.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-03T06:09:15.810 INFO:tasks.workunit.client.0.smithi039.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-03T06:09:15.810 INFO:tasks.workunit.client.0.smithi039.stderr: else f 2024-06-03T06:09:15.810 INFO:tasks.workunit.client.0.smithi039.stderr: end; 2024-06-03T06:09:15.810 INFO:tasks.workunit.client.0.smithi039.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-03T06:09:15.810 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-03T06:09:15.810 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-03T06:09:15.810 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-03T06:09:15.811 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-03T06:09:15.811 INFO:tasks.workunit.client.0.smithi039.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-03T06:09:15.815 INFO:tasks.workunit.client.0.smithi039.stderr:+ main osd-scrub-repair 2024-06-03T06:09:15.815 INFO:tasks.workunit.client.0.smithi039.stderr:+ local dir=td/osd-scrub-repair 2024-06-03T06:09:15.816 INFO:tasks.workunit.client.0.smithi039.stderr:+ shift 2024-06-03T06:09:15.816 INFO:tasks.workunit.client.0.smithi039.stderr:+ shopt -s -o xtrace 2024-06-03T06:09:15.816 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-03T06:09:15.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-06-03T06:09:15.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-06-03T06:09:15.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-06-03T06:09:15.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-06-03T06:09:15.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-06-03T06:09:15.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-06-03T06:09:15.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-06-03T06:09:15.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-06-03T06:09:15.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-repair 2024-06-03T06:09:15.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-06-03T06:09:15.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-06-03T06:09:15.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-06-03T06:09:15.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-06-03T06:09:15.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-06-03T06:09:15.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-06-03T06:09:15.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none ' 2024-06-03T06:09:15.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-06-03T06:09:15.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-06-03T06:09:15.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-06-03T06:09:15.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-06-03T06:09:15.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-03T06:09:15.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-06-03T06:09:15.820 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_auto_repair_bluestore_basic 2024-06-03T06:09:15.820 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_auto_repair_bluestore_failed 2024-06-03T06:09:15.820 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-06-03T06:09:15.820 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_auto_repair_bluestore_scrub 2024-06-03T06:09:15.820 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_auto_repair_bluestore_tag 2024-06-03T06:09:15.820 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_auto_repair_erasure_coded_appends 2024-06-03T06:09:15.820 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-06-03T06:09:15.820 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-06-03T06:09:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-06-03T06:09:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_corrupt_and_repair_lrc_appends 2024-06-03T06:09:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-06-03T06:09:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_corrupt_and_repair_replicated 2024-06-03T06:09:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_corrupt_scrub_erasure_appends 2024-06-03T06:09:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-06-03T06:09:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_corrupt_scrub_replicated 2024-06-03T06:09:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_corrupt_snapset_scrub_rep 2024-06-03T06:09:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_list_missing_erasure_coded_appends 2024-06-03T06:09:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_list_missing_erasure_coded_overwrites 2024-06-03T06:09:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_periodic_scrub_replicated 2024-06-03T06:09:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_repair_stats 2024-06-03T06:09:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_repair_stats_ec 2024-06-03T06:09:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_request_scrub_priority 2024-06-03T06:09:15.822 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_scrub_warning 2024-06-03T06:09:15.822 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_skip_non_repair_during_recovery 2024-06-03T06:09:15.822 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_unfound_erasure_coded_appends 2024-06-03T06:09:15.822 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_unfound_erasure_coded_overwrites' 2024-06-03T06:09:15.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T06:09:15.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T06:09:15.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T06:09:15.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T06:09:15.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:09:15.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:09:15.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:09:15.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:09:15.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:09:15.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:09:15.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:09:15.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:09:15.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:09:15.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:09:15.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:09:15.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:09:15.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:09:15.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:09:15.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:09:15.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:09:15.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:09:15.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:09:15.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:09:15.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:09:15.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:09:15.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:09:15.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:09:15.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:09:15.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:09:15.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:09:15.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:09:15.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:09:15.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T06:09:15.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:09:15.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:09:15.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:09:15.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T06:09:15.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:09:15.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-03T06:09:15.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-03T06:09:15.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:09:15.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T06:09:15.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-06-03T06:09:15.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-06-03T06:09:15.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-06-03T06:09:15.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-06-03T06:09:15.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T06:09:15.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T06:09:15.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T06:09:15.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T06:09:15.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T06:09:15.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-06-03T06:09:16.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T06:09:16.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:09:16.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:09:16.192 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:09:16.193 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:09:16.193 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:09:16.193 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:09:16.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-03T06:09:16.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T06:09:16.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T06:09:16.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:09:16.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:09:16.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T06:09:16.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T06:09:16.386 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:09:16.386 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:09:16.386 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:09:16.387 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:09:16.387 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:09:16.387 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:09:16.387 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:09:16.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:09:16.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T06:09:16.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T06:09:16.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:09:16.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:09:16.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T06:09:16.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T06:09:16.470 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:09:16.470 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:09:16.470 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:09:16.470 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:09:16.471 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:09:16.471 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:09:16.471 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:09:16.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:09:16.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T06:09:16.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-06-03T06:09:16.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T06:09:16.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T06:09:16.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T06:09:16.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T06:09:16.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T06:09:16.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T06:09:17.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T06:09:17.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:09:17.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:09:17.362 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:09:17.362 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:09:17.362 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:09:17.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:09:17.363 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:09:17.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:09:17.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-06-03T06:09:17.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:09:17.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:09:17.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T06:09:17.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:09:17.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:09:17.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:09:17.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:09:17.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:09:17.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:09:17.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:09:17.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:09:17.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:09:17.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:09:17.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:09:17.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:09:17.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:09:17.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:09:17.395 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:09:17.395 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:09:17.395 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:09:17.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:09:17.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:09:17.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:09:17.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:09:17.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:09:17.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:09:17.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:09:17.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:09:17.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:09:17.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:09:17.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:09:17.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:09:17.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-06-03T06:09:17.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:09:17.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:09:17.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=585ad79c-3099-4757-8d93-6f5e8f726ba6 2024-06-03T06:09:17.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 585ad79c-3099-4757-8d93-6f5e8f726ba6' 2024-06-03T06:09:17.397 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 585ad79c-3099-4757-8d93-6f5e8f726ba6 2024-06-03T06:09:17.398 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:09:17.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANXl1mZ/KrGBAAZxwSUAJzNArjVQVTZAKV2w== 2024-06-03T06:09:17.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANXl1mZ/KrGBAAZxwSUAJzNArjVQVTZAKV2w=="}' 2024-06-03T06:09:17.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 585ad79c-3099-4757-8d93-6f5e8f726ba6 -i td/osd-scrub-repair/0/new.json 2024-06-03T06:09:17.627 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:09:17.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T06:09:17.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQANXl1mZ/KrGBAAZxwSUAJzNArjVQVTZAKV2w== --osd-uuid 585ad79c-3099-4757-8d93-6f5e8f726ba6 2024-06-03T06:09:17.670 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:17.670+0000 7f9d49399640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:09:17.678 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:17.678+0000 7f9d49399640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:09:17.679 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:17.679+0000 7f9d49399640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:09:17.680 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:17.680+0000 7f9d49399640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:09:17.680 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:17.680+0000 7f9d49399640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:09:17.680 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:17.680+0000 7f9d49399640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:09:17.761 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:17.761+0000 7f9d49399640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T06:09:23.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T06:09:23.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:09:23.499 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:09:23.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T06:09:23.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:09:23.877 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:09:23.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T06:09:23.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-06-03T06:09:23.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:09:23.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:09:23.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:09:23.906 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:23.906+0000 7f07d738e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:09:23.908 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:23.908+0000 7f07d738e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:09:23.909 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:23.910+0000 7f07d738e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:09:24.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T06:09:24.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:09:24.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:09:24.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:09:24.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:09:24.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:24.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:09:24.166 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:09:24.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:24.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:24.422 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:24.423+0000 7f07d738e640 -1 Falling back to public interface 2024-06-03T06:09:24.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:25.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:25.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:25.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:09:25.453 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:09:25.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:25.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:25.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:26.745 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:26.744+0000 7f07d738e640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:09:26.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:26.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:26.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:09:26.759 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:09:26.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:26.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:27.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:28.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:28.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:28.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:09:28.215 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:09:28.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:28.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:28.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:29.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:29.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:29.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:09:29.570 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:09:29.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:29.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:29.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:30.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:30.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:30.882 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:09:30.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:09:30.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:30.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:31.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:32.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:32.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:32.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:09:32.183 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:09:32.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:32.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:32.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:33.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:33.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:33.562 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:09:33.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:09:33.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:33.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:33.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:34.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:34.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:34.862 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:09:34.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:09:34.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:34.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:35.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:36.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:36.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:36.166 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:09:36.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:09:36.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:36.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:36.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:37.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:37.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:37.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:09:37.541 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:09:37.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:37.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:37.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:38.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:38.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:38.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:09:38.840 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:09:38.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:38.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:39.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:40.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:40.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:40.503 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:09:40.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:09:40.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:40.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:40.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:41.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:41.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:41.991 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:09:41.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:09:41.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:41.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:42.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:43.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:43.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:43.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:09:43.684 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:09:43.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:43.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:44.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:45.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:45.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:45.057 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:09:45.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:09:45.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:45.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:45.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:46.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:46.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:46.423 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:09:46.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:09:46.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:46.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:46.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:47.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:47.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:47.790 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:09:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:09:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:48.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:49.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:49.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:49.137 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:09:49.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:09:49.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:49.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:09:49.436 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2012602064,v1:127.0.0.1:6803/2012602064] [v2:127.0.0.1:6804/2012602064,v1:127.0.0.1:6805/2012602064] exists,up 585ad79c-3099-4757-8d93-6f5e8f726ba6 2024-06-03T06:09:49.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:09:49.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:09:49.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:09:49.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-06-03T06:09:49.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:09:49.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:09:49.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T06:09:49.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:09:49.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:09:49.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:09:49.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:09:49.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:09:49.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:09:49.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:09:49.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:09:49.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:09:49.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:09:49.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:09:49.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:09:49.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:09:49.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:09:49.442 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:09:49.442 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:09:49.442 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:09:49.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:09:49.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:09:49.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:09:49.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:09:49.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:09:49.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:09:49.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:09:49.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:09:49.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:09:49.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:09:49.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:09:49.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:09:49.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-06-03T06:09:49.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:09:49.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:09:49.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2c5df2c9-d3c4-43da-a4bf-4d1c71199212 2024-06-03T06:09:49.443 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 2c5df2c9-d3c4-43da-a4bf-4d1c71199212 2024-06-03T06:09:49.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2c5df2c9-d3c4-43da-a4bf-4d1c71199212' 2024-06-03T06:09:49.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:09:49.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAtXl1mFBldGxAAdIPcMEw9f0PHQdTVPXLM+A== 2024-06-03T06:09:49.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAtXl1mFBldGxAAdIPcMEw9f0PHQdTVPXLM+A=="}' 2024-06-03T06:09:49.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2c5df2c9-d3c4-43da-a4bf-4d1c71199212 -i td/osd-scrub-repair/1/new.json 2024-06-03T06:09:49.837 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:09:49.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T06:09:49.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQAtXl1mFBldGxAAdIPcMEw9f0PHQdTVPXLM+A== --osd-uuid 2c5df2c9-d3c4-43da-a4bf-4d1c71199212 2024-06-03T06:09:49.881 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:49.881+0000 7f4fa5a94640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:09:49.913 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:49.913+0000 7f4fa5a94640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:09:49.914 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:49.915+0000 7f4fa5a94640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:09:49.915 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:49.915+0000 7f4fa5a94640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:09:49.915 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:49.915+0000 7f4fa5a94640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:09:49.915 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:49.915+0000 7f4fa5a94640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:09:49.997 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:49.997+0000 7f4fa5a94640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T06:09:55.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T06:09:55.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:09:55.965 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:09:55.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T06:09:55.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:09:56.341 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:09:56.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T06:09:56.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-06-03T06:09:56.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:09:56.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:09:56.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:09:56.371 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:56.371+0000 7f00d5c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:09:56.373 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:56.373+0000 7f00d5c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:09:56.374 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:56.374+0000 7f00d5c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:09:56.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T06:09:56.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:09:56.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:09:56.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:09:56.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:09:56.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:56.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:09:56.644 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:09:56.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:56.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:09:56.889 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:56.890+0000 7f00d5c0f640 -1 Falling back to public interface 2024-06-03T06:09:56.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:57.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:57.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:57.932 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:09:57.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:09:57.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:57.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:09:58.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:09:59.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:09:59.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:09:59.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:09:59.237 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:09:59.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:09:59.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:09:59.257 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:09:59.257+0000 7f00d5c0f640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:09:59.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:00.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:00.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:00.583 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:10:00.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:10:00.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:00.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:01.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:02.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:02.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:02.059 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:10:02.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:10:02.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:02.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:02.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:03.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:03.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:03.358 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:10:03.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:10:03.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:03.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:03.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:04.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:04.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:04.677 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:10:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:10:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:05.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:06.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:06.085 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:10:06.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:06.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:10:06.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:06.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:06.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:07.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:07.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:07.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:10:07.393 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:10:07.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:07.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:07.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:08.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:08.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:08.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:10:08.693 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:10:08.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:08.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:09.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:10.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:10.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:10.079 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:10:10.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:10:10.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:10.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:10.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:11.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:11.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:11.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:10:11.383 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:10:11.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:11.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:11.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:12.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:12.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:12.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:10:12.787 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:10:12.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:12.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:13.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:14.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:14.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:14.367 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:10:14.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:10:14.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:14.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:14.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:15.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:15.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:15.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:10:15.894 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:10:15.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:15.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:16.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:17.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:17.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:17.303 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:10:17.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:10:17.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:17.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:18.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:19.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:19.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:19.333 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:10:19.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:10:19.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:19.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:19.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:20.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:20.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:20.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:10:20.835 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:10:20.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:20.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:21.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:22.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:22.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:22.516 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:10:22.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:10:22.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:22.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:22.834 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3772624061,v1:127.0.0.1:6811/3772624061] [v2:127.0.0.1:6812/3772624061,v1:127.0.0.1:6813/3772624061] exists,up 2c5df2c9-d3c4-43da-a4bf-4d1c71199212 2024-06-03T06:10:22.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:10:22.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:10:22.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:10:22.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-06-03T06:10:22.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T06:10:23.114 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-06-03T06:10:23.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T06:10:23.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T06:10:23.517 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T06:10:23.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:10:24.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T06:10:26.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-06-03T06:10:26.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:10:26.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:10:26.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:10:26.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:10:26.505 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:10:26.505 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:10:26.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:10:26.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:10:26.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:10:26.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:10:26.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:10:26.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:10:26.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:10:26.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:10:26.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:10:26.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:10:26.899 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T06:10:26.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:10:26.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:10:26.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:10:27.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-06-03T06:10:27.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-06-03T06:10:27.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-06-03T06:10:27.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:10:27.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:10:27.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-06-03T06:10:27.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-06-03T06:10:27.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966' 2024-06-03T06:10:27.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:10:27.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-06-03T06:10:27.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:10:27.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:10:27.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-06-03T06:10:27.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:10:27.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-06-03T06:10:27.118 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836489 2024-06-03T06:10:27.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-06-03T06:10:27.118 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:10:27.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836489 2024-06-03T06:10:27.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:10:28.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:10:28.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:10:28.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-06-03T06:10:28.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:10:28.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-06-03T06:10:28.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:10:28.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:10:28.713 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-06-03T06:10:28.713 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:10:28.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-06-03T06:10:28.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-06-03T06:10:28.714 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672966 2024-06-03T06:10:28.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:10:29.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672966 2024-06-03T06:10:29.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:10:29.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:10:29.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:10:29.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T06:10:29.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:10:29.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:10:29.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:10:29.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:10:29.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:10:29.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:10:29.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:10:29.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:10:29.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:10:29.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:10:29.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:10:30.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T06:10:30.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:10:30.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:10:30.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-06-03T06:10:30.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T06:10:30.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-06-03T06:10:30.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-03T06:10:30.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T06:10:30.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T06:10:30.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T06:10:30.408 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T06:10:30.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T06:10:30.910 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T06:10:30.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T06:10:30.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T06:10:30.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-03T06:10:31.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-06-03T06:10:31.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-06-03T06:10:31.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:10:31.157 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-06-03T06:10:31.157 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-06-03T06:10:31.157 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:10:31.157 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-06-03T06:10:31.157 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:10:31.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T06:10:31.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-06-03T06:10:31.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T06:10:31.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-06-03T06:10:31.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-03T06:10:31.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-06-03T06:10:31.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-06-03T06:10:31.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-06-03T06:10:31.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:10:31.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:10:31.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:10:31.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:10:31.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-06-03T06:10:31.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:10:31.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:10:31.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:10:31.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:10:31.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:10:31.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:10:31.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:10:31.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:10:31.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:10:31.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:10:33.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:10:33.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-06-03T06:10:33.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:10:33.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:10:33.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:10:33.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:10:33.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:10:33.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-06-03T06:10:34.277 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-03T06:10:34.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T06:10:34.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:10:34.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:10:34.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:10:34.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:10:34.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:10:34.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:10:34.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:10:34.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:10:34.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:10:34.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:10:34.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:10:34.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:10:34.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:10:34.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:10:34.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:10:34.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:10:34.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:10:34.796 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:10:34.797 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:10:34.797 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:10:34.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:10:34.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:10:34.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:10:34.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:10:34.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:10:34.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:10:34.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:10:34.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:10:34.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:10:34.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:10:34.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:10:34.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:10:34.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:10:34.798 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:10:34.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:10:34.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:10:34.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:10:34.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:10:34.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:10:34.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:10:34.826 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:10:34.826+0000 7f4c5570a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:10:34.829 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:10:34.829+0000 7f4c5570a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:10:34.830 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:10:34.830+0000 7f4c5570a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:10:35.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:10:35.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:10:35.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:10:35.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:10:35.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:10:35.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:35.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:10:35.100 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:10:35.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:35.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:10:35.346 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:10:35.346+0000 7f4c5570a640 -1 Falling back to public interface 2024-06-03T06:10:35.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:36.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:36.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:36.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:10:36.389 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:10:36.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:36.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:10:36.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:37.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:37.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:37.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:10:37.704 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:10:37.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:37.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:10:37.812 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:10:37.812+0000 7f4c5570a640 -1 osd.0 22 log_to_monitors true 2024-06-03T06:10:38.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:39.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:39.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:39.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:10:39.031 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:10:39.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:39.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:10:39.331 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/3837632222,v1:127.0.0.1:6803/3837632222] [v2:127.0.0.1:6804/3837632222,v1:127.0.0.1:6805/3837632222] exists,up 585ad79c-3099-4757-8d93-6f5e8f726ba6 2024-06-03T06:10:39.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:10:39.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:10:39.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:10:39.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:10:39.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:10:39.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:10:39.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:10:39.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:10:39.332 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:10:39.332 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:10:39.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:10:39.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:10:39.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:10:39.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:10:39.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:10:39.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:10:39.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:10:39.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:10:39.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:10:39.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:10:39.787 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T06:10:39.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:10:39.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:10:39.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:10:39.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-06-03T06:10:39.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-06-03T06:10:39.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-06-03T06:10:39.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:10:39.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:10:40.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-06-03T06:10:40.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-06-03T06:10:40.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672970' 2024-06-03T06:10:40.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:10:40.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-06-03T06:10:40.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:10:40.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:10:40.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-06-03T06:10:40.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:10:40.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-06-03T06:10:40.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-06-03T06:10:40.008 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 111669149698 2024-06-03T06:10:40.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:10:40.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-06-03T06:10:40.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:10:41.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:10:41.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:10:41.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-06-03T06:10:41.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:10:42.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T06:10:42.605 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:10:42.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-06-03T06:10:42.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:10:42.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-06-03T06:10:42.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:10:42.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:10:42.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-06-03T06:10:42.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:10:42.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-06-03T06:10:42.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-06-03T06:10:42.910 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672970 2024-06-03T06:10:42.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:10:43.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-06-03T06:10:43.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:10:43.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:10:43.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:10:43.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T06:10:43.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:10:43.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:10:43.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:10:43.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:10:43.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:10:43.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:10:43.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:10:43.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:10:43.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:10:43.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:10:43.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:10:44.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T06:10:44.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:10:44.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:10:44.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-06-03T06:10:44.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-06-03T06:10:44.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:10:44.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-06-03T06:10:44.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:10:44.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-06-03T06:10:44.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-06-03T06:10:44.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-06-03T06:10:44.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-06-03T06:10:44.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:10:44.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:10:44.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:10:44.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:10:44.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:10:23.463306+0000 2024-06-03T06:10:44.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-06-03T06:10:45.065 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 1.3 on osd.1 to repair 2024-06-03T06:10:45.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-06-03T06:10:23.463306+0000 2024-06-03T06:10:45.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-06-03T06:10:45.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:10:23.463306+0000 2024-06-03T06:10:45.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:10:45.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:10:45.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:10:45.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:10:45.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:10:45.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:10:45.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:10:45.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:10:45.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:10:23.463306+0000 '>' 2024-06-03T06:10:23.463306+0000 2024-06-03T06:10:45.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:10:46.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:10:46.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:10:46.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:10:46.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:10:46.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:10:46.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:10:46.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:10:46.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:10:23.463306+0000 '>' 2024-06-03T06:10:23.463306+0000 2024-06-03T06:10:46.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:10:47.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:10:47.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:10:47.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:10:47.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:10:47.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:10:47.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:10:47.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:10:47.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:10:23.463306+0000 '>' 2024-06-03T06:10:23.463306+0000 2024-06-03T06:10:47.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:10:48.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:10:48.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:10:48.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:10:48.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:10:48.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:10:48.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:10:48.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:10:49.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:10:23.463306+0000 '>' 2024-06-03T06:10:23.463306+0000 2024-06-03T06:10:49.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:10:50.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:10:50.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:10:50.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:10:50.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:10:50.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:10:50.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:10:50.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:10:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:10:45.722895+0000 '>' 2024-06-03T06:10:23.463306+0000 2024-06-03T06:10:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:10:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:10:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:10:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:10:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:10:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:10:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:10:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:10:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:10:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:10:50.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:10:50.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:10:50.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:10:50.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:10:50.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:10:50.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:10:50.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:10:50.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:10:50.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:10:50.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:10:50.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:10:50.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:10:50.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:10:50.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:10:50.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-03T06:10:52.291 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-03T06:10:52.291 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:10:52.291 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:10:52.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:10:52.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:10:52.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:10:52.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:10:52.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:10:52.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:10:52.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:10:52.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:10:52.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:10:52.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:10:52.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:10:52.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:10:52.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:10:52.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:10:52.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:10:52.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:10:52.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:10:52.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:10:52.572 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:10:52.572 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:10:52.572 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:10:52.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:10:52.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:10:52.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:10:52.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:10:52.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:10:52.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:10:52.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:10:52.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:10:52.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:10:52.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:10:52.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:10:52.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:10:52.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:10:52.574 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:10:52.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:10:52.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:10:52.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:10:52.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:10:52.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:10:52.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:10:52.602 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:10:52.602+0000 7fa896611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:10:52.604 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:10:52.605+0000 7fa896611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:10:52.606 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:10:52.606+0000 7fa896611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:10:52.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:10:52.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:10:52.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:10:52.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:10:52.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:10:52.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:52.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:10:52.873 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:10:52.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:52.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:53.121 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:10:53.121+0000 7fa896611640 -1 Falling back to public interface 2024-06-03T06:10:53.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:54.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:54.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:54.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:10:54.162 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:10:54.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:54.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:54.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:10:55.354 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:10:55.354+0000 7fa896611640 -1 osd.1 28 log_to_monitors true 2024-06-03T06:10:55.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:10:55.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:10:55.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:10:55.461 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:10:55.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:10:55.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:10:55.787 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 32 up_thru 27 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/828497292,v1:127.0.0.1:6811/828497292] [v2:127.0.0.1:6812/828497292,v1:127.0.0.1:6813/828497292] exists,up 2c5df2c9-d3c4-43da-a4bf-4d1c71199212 2024-06-03T06:10:55.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:10:55.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:10:55.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:10:55.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:10:55.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:10:55.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:10:55.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:10:55.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:10:55.788 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:10:55.788 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:10:55.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:10:55.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:10:55.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:10:55.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:10:55.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:10:55.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:10:55.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:10:55.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:10:55.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:10:56.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:10:56.169 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T06:10:56.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:10:56.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:10:56.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:10:56.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-06-03T06:10:56.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-06-03T06:10:56.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702' 2024-06-03T06:10:56.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:10:56.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:10:56.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-06-03T06:10:56.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-06-03T06:10:56.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-06-03T06:10:56.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:10:56.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149702 2024-06-03T06:10:56.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:10:56.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:10:56.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149702 2024-06-03T06:10:56.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:10:56.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-06-03T06:10:56.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-06-03T06:10:56.412 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 111669149702 2024-06-03T06:10:56.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:10:56.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149701 -lt 111669149702 2024-06-03T06:10:56.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:10:57.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:10:57.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:10:58.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149702 -lt 111669149702 2024-06-03T06:10:58.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:10:58.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-06-03T06:10:58.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:10:58.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:10:58.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-06-03T06:10:58.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:10:58.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-06-03T06:10:58.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-06-03T06:10:58.078 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 137438953474 2024-06-03T06:10:58.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:10:58.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-06-03T06:10:58.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:10:58.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:10:58.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:10:58.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T06:10:58.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:10:58.730 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:10:58.730 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:10:58.730 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:10:58.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:10:58.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:10:58.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:10:59.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-03T06:10:59.019 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:10:59.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:10:59.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:10:59.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 4 2024-06-03T06:10:59.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-03T06:10:59.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:10:59.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-03T06:10:59.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:10:59.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:10:59.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:10:59.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:10:59.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:10:59.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:10:59.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:10:59.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:10:59.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:10:59.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:10:59.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:10:59.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:10:59.777 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:10:59.777 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:11:00.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T06:11:00.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:11:00.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:11:00.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:11:00.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:11:00.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T06:11:00.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:11:00.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:11:00.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:11:00.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:11:00.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:11:00.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:11:00.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:11:00.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:11:00.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:11:00.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:11:00.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:11:00.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:11:00.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:11:00.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:11:00.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:11:00.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:11:00.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:11:00.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:11:00.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:11:00.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:11:00.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:11:00.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:11:00.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:11:00.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:11:00.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:11:00.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:11:00.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:11:00.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:11:00.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T06:11:00.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T06:11:00.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T06:11:00.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T06:11:00.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:11:00.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:11:00.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:11:00.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:11:00.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:11:00.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:11:00.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:11:00.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:11:00.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:11:00.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:11:00.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:11:00.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:11:00.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:11:00.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:11:00.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:11:00.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:11:00.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:11:00.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:11:00.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:11:00.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:11:00.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:11:00.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:11:00.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:11:00.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:11:00.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:11:00.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:11:00.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:11:00.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:11:00.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T06:11:00.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:11:00.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:11:00.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:11:00.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T06:11:00.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:11:00.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T06:11:00.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:11:00.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T06:11:00.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-06-03T06:11:00.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-06-03T06:11:00.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-06-03T06:11:00.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-06-03T06:11:00.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-06-03T06:11:00.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-06-03T06:11:00.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-06-03T06:11:00.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-06-03T06:11:00.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-06-03T06:11:00.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=1 2024-06-03T06:11:00.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=testpool 2024-06-03T06:11:00.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=testpool 2024-06-03T06:11:00.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-06-03T06:11:00.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-06-03T06:11:00.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-06-03T06:11:00.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-06-03T06:11:01.943 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 7220952b-1537-4720-8cac-2770e07493b3 2024-06-03T06:11:02.213 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:11:02.259 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:02.259+0000 7ff3b8220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:11:02.272 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:02.273+0000 7ff3b8220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:11:02.273 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:02.274+0000 7ff3b8220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:11:02.274 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:02.274+0000 7ff3b8220640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:11:02.274 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:02.274+0000 7ff3b8220640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:11:02.274 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:02.274+0000 7ff3b8220640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:11:02.365 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:02.365+0000 7ff3b8220640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T06:11:07.872 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:11:08.276 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:11:08.308 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:08.308+0000 7fd072622640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:11:08.310 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:08.311+0000 7fd072622640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:11:08.311 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:08.312+0000 7fd072622640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:11:08.563 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:11:08.826 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:08.826+0000 7fd072622640 -1 Falling back to public interface 2024-06-03T06:11:09.848 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:11:11.146 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:11:11.147 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:11.147+0000 7fd072622640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:11:12.590 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:11:13.895 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:11:15.199 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:11:16.502 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:11:17.807 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:11:19.129 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:11:20.428 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:11:21.728 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:11:23.097 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:11:24.509 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:11:26.101 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:11:27.831 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:11:29.208 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:11:30.731 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:11:32.340 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:11:33.808 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:11:34.126 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3066794153,v1:127.0.0.1:6803/3066794153] [v2:127.0.0.1:6804/3066794153,v1:127.0.0.1:6805/3066794153] exists,up 7220952b-1537-4720-8cac-2770e07493b3 2024-06-03T06:11:34.130 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 9d593ca3-b014-4a9e-a1e3-a0cbb3d763f4 2024-06-03T06:11:34.509 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:11:34.557 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:34.557+0000 7f9f83c98640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:11:34.602 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:34.602+0000 7f9f83c98640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:11:34.603 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:34.603+0000 7f9f83c98640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:11:34.603 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:34.603+0000 7f9f83c98640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:11:34.603 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:34.603+0000 7f9f83c98640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:11:34.603 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:34.603+0000 7f9f83c98640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:11:34.694 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:34.694+0000 7f9f83c98640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T06:11:40.761 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:11:41.134 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:11:41.167 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:41.167+0000 7fb5e976e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:11:41.176 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:41.176+0000 7fb5e976e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:11:41.177 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:41.178+0000 7fb5e976e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:11:41.457 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:11:41.714 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:41.713+0000 7fb5e976e640 -1 Falling back to public interface 2024-06-03T06:11:42.746 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:11:44.037 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:11:44.037+0000 7fb5e976e640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:11:44.050 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:11:45.350 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:11:46.659 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:11:48.066 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:11:49.372 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:11:50.679 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:11:52.111 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:11:53.412 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:11:54.716 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:11:56.095 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:11:57.400 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:11:58.996 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:12:00.408 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:12:02.212 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:12:04.172 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:12:05.615 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:12:07.197 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:12:07.505 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3084834936,v1:127.0.0.1:6811/3084834936] [v2:127.0.0.1:6812/3084834936,v1:127.0.0.1:6813/3084834936] exists,up 9d593ca3-b014-4a9e-a1e3-a0cbb3d763f4 2024-06-03T06:12:07.509 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 35b2d506-f493-46b6-8615-898377543ed5 2024-06-03T06:12:07.887 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:12:07.935 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:07.934+0000 7f6a8bcac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:12:07.972 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:07.971+0000 7f6a8bcac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:12:07.973 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:07.972+0000 7f6a8bcac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:12:07.973 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:07.972+0000 7f6a8bcac640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:12:07.973 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:07.972+0000 7f6a8bcac640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:12:07.973 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:07.972+0000 7f6a8bcac640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:12:08.038 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:08.038+0000 7f6a8bcac640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T06:12:14.354 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T06:12:14.730 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:12:14.764 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:14.764+0000 7fd442eae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:12:14.766 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:14.766+0000 7fd442eae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:12:14.767 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:14.767+0000 7fd442eae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:12:15.033 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:12:15.305 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:15.306+0000 7fd442eae640 -1 Falling back to public interface 2024-06-03T06:12:16.322 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:12:17.622 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:12:17.641 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:17.641+0000 7fd442eae640 -1 osd.2 0 log_to_monitors true 2024-06-03T06:12:18.950 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:12:20.299 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:12:21.715 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:12:23.018 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:12:24.450 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:12:25.751 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:12:27.053 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:12:28.355 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:12:29.701 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:12:31.003 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:12:32.623 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:12:34.504 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:12:36.284 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:12:37.582 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:12:39.436 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:12:41.105 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:12:41.405 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/869260514,v1:127.0.0.1:6819/869260514] [v2:127.0.0.1:6820/869260514,v1:127.0.0.1:6821/869260514] exists,up 35b2d506-f493-46b6-8615-898377543ed5 2024-06-03T06:12:41.754 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'testpool' created 2024-06-03T06:12:43.519 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836500 2024-06-03T06:12:45.111 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672974 2024-06-03T06:12:45.403 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509442 2024-06-03T06:12:47.999 INFO:tasks.workunit.client.0.smithi039.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-06-03T06:12:47.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-06-03T06:12:48.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-06-03T06:12:48.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-06-03T06:12:48.379 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 2 2024-06-03T06:12:48.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-06-03T06:12:48.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:12:48.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:12:48.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:12:48.392 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:12:48.392 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:12:48.392 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:12:48.392 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:12:48.392 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:12:48.393 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:12:48.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:12:48.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:12:48.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:12:48.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:12:48.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:12:48.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:12:48.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:12:48.869 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:12:48.869 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:12:48.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:12:48.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:12:48.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:12:49.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-06-03T06:12:49.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-06-03T06:12:49.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-06-03T06:12:49.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:12:49.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:12:49.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-06-03T06:12:49.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-06-03T06:12:49.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672976' 2024-06-03T06:12:49.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:12:49.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:12:49.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-06-03T06:12:49.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-06-03T06:12:49.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672976 2-64424509444' 2024-06-03T06:12:49.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:12:49.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-06-03T06:12:49.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:12:49.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:12:49.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-06-03T06:12:49.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:12:49.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-06-03T06:12:49.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-06-03T06:12:49.249 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836503 2024-06-03T06:12:49.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:12:49.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836503 2024-06-03T06:12:49.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:12:50.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:12:50.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:12:50.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-06-03T06:12:50.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:12:50.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-06-03T06:12:50.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:12:50.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:12:50.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-06-03T06:12:50.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:12:50.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-06-03T06:12:50.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-06-03T06:12:50.843 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672976 2024-06-03T06:12:50.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:12:51.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-06-03T06:12:51.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:12:51.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-06-03T06:12:51.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:12:51.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:12:51.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-06-03T06:12:51.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:12:51.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-06-03T06:12:51.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-06-03T06:12:51.159 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509444 2024-06-03T06:12:51.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:12:51.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509444 2024-06-03T06:12:51.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:12:51.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:12:51.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:12:51.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:12:51.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:12:51.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:12:51.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:12:51.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:12:51.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:12:51.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:12:51.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:12:52.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T06:12:52.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:12:52.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:12:52.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:12:52.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-03T06:12:52.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-03T06:12:52.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:12:52.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-03T06:12:52.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:12:52.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:12:52.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:12:52.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:12:52.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:12:52.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:12:52.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:12:52.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:12:52.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:12:52.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:12:52.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T06:12:52.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:12:52.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:12:52.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:12:53.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-03T06:12:53.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-03T06:12:53.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:12:53.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:12:53.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:12:53.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:12:53.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:12:53.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:12:53.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:12:53.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T06:12:53.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T06:12:53.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T06:12:53.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:12:53.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T06:12:53.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:12:53.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:12:53.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:12:53.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:12:53.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:12:53.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:12:53.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:12:53.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:12:54.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T06:12:54.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:12:54.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:12:54.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:12:54.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-03T06:12:54.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-03T06:12:54.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:12:54.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:12:54.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:12:54.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:12:54.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:12:54.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:12:54.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:12:54.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T06:12:54.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T06:12:54.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-03T06:12:54.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:12:54.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-03T06:12:55.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:12:55.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:12:55.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:12:55.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:12:55.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:12:55.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:12:55.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:12:55.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:12:55.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:12:55.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:12:55.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:12:55.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:12:55.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:12:55.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:12:55.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:12:55.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-06-03T06:12:55.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-06-03T06:12:55.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-03T06:12:55.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-06-03T06:12:55.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-03T06:12:55.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:12:55.984 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-03T06:12:55.984 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:12:55.984 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:12:55.984 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:12:55.984 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:12:56.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T06:12:56.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:12:56.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T06:12:56.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-06-03T06:12:56.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:12:56.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:12:56.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:12:56.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:12:56.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-06-03T06:12:56.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:12:56.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:12:56.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:12:56.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:12:56.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:12:56.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:12:56.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:12:56.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:12:56.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:12:56.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:12:56.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:12:56.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-06-03T06:12:56.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:12:56.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:12:56.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:12:56.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:12:56.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:12:56.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-06-03T06:12:58.310 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-03T06:12:58.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T06:12:58.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:12:58.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:12:58.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:12:58.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:12:58.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:12:58.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:12:58.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:12:58.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:12:58.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:12:58.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:12:58.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:12:58.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:12:58.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:12:58.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:12:58.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:12:58.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:12:58.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:12:58.829 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:12:58.829 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:12:58.829 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:12:58.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:12:58.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:12:58.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:12:58.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:12:58.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:12:58.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:12:58.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:12:58.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:12:58.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:12:58.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:12:58.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:12:58.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:12:58.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:12:58.831 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:12:58.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:12:58.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:12:58.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:12:58.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:12:58.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:12:58.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:12:58.859 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:58.858+0000 7f0178a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:12:58.862 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:58.861+0000 7f0178a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:12:58.863 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:58.862+0000 7f0178a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:12:59.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:12:59.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:12:59.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:12:59.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:12:59.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:12:59.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:12:59.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:12:59.132 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:12:59.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:12:59.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:12:59.378 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:12:59.377+0000 7f0178a0c640 -1 Falling back to public interface 2024-06-03T06:12:59.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:00.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:00.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:00.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:13:00.423 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:13:00.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:00.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:00.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:01.709 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:01.709+0000 7f0178a0c640 -1 osd.0 20 log_to_monitors true 2024-06-03T06:13:01.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:01.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:01.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:13:01.722 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:13:01.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:01.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:02.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:03.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:03.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:03.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:13:03.048 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:13:03.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:03.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:03.346 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/3573732299,v1:127.0.0.1:6803/3573732299] [v2:127.0.0.1:6804/3573732299,v1:127.0.0.1:6805/3573732299] exists,up 7220952b-1537-4720-8cac-2770e07493b3 2024-06-03T06:13:03.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:13:03.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:13:03.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:13:03.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:13:03.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:13:03.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:13:03.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:13:03.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:13:03.347 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:13:03.347 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:13:03.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:13:03.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:13:03.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:13:03.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:13:03.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:13:03.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:13:03.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:13:03.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:13:03.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:13:03.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:13:03.726 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:13:03.726 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:13:03.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:13:03.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:13:03.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:13:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215106 2024-06-03T06:13:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215106 2024-06-03T06:13:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106' 2024-06-03T06:13:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:13:03.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:13:03.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-06-03T06:13:03.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-06-03T06:13:03.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672980' 2024-06-03T06:13:03.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:13:03.944 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:13:04.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-06-03T06:13:04.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-06-03T06:13:04.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672980 2-64424509448' 2024-06-03T06:13:04.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:13:04.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215106 2024-06-03T06:13:04.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:13:04.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:13:04.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215106 2024-06-03T06:13:04.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:13:04.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215106 2024-06-03T06:13:04.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-06-03T06:13:04.054 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 103079215106 2024-06-03T06:13:04.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:13:04.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215106 -lt 103079215106 2024-06-03T06:13:04.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:13:04.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-06-03T06:13:04.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:13:04.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:13:04.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-06-03T06:13:04.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:13:04.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-06-03T06:13:04.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-06-03T06:13:04.347 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672980 2024-06-03T06:13:04.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:13:04.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-06-03T06:13:04.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:13:04.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-06-03T06:13:04.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:13:04.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:13:04.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-06-03T06:13:04.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:13:04.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-06-03T06:13:04.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-06-03T06:13:04.656 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509448 2024-06-03T06:13:04.657 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:13:04.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-06-03T06:13:04.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:13:04.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:13:04.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:13:05.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:13:05.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:13:05.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:13:05.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:13:05.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:13:05.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:13:05.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:13:05.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:13:05.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:13:05.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:13:05.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:13:05.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:13:05.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:13:05.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:13:05.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:13:05.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-06-03T06:13:06.074 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: { 2024-06-03T06:13:06.074 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-06-03T06:13:06.074 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:13:06.089 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: { 2024-06-03T06:13:06.089 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-06-03T06:13:06.089 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:13:06.102 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: { 2024-06-03T06:13:06.102 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-06-03T06:13:06.102 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:13:06.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-06-03T06:13:06.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-03T06:13:06.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:13:06.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-06-03T06:13:06.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:13:06.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-06-03T06:13:06.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-06-03T06:13:06.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:13:06.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:13:06.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:13:06.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:13:06.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-06-03T06:13:06.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-06-03T06:13:06.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:13:06.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:13:06.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:13:06.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:13:07.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-06-03T06:12:41.705047+0000 2024-06-03T06:13:07.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-06-03T06:13:07.169 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T06:13:07.169 INFO:tasks.workunit.client.0.smithi039.stdout: "deep": true, 2024-06-03T06:13:07.170 INFO:tasks.workunit.client.0.smithi039.stdout: "must": false, 2024-06-03T06:13:07.170 INFO:tasks.workunit.client.0.smithi039.stdout: "stamp": "2024-05-20T06:11:27.170489+0000" 2024-06-03T06:13:07.170 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:13:07.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-06-03T06:12:41.705047+0000 2024-06-03T06:13:07.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-03T06:13:07.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:12:41.705047+0000 2024-06-03T06:13:07.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:13:07.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:13:07.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:13:07.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:13:07.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:13:07.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:13:07.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:13:07.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:13:07.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:12:41.705047+0000 '>' 2024-06-03T06:12:41.705047+0000 2024-06-03T06:13:07.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:13:08.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:13:08.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:13:08.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:13:08.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:13:08.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:13:08.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:13:08.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:13:08.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-05-20T06:11:27.170489+0000 '>' 2024-06-03T06:12:41.705047+0000 2024-06-03T06:13:08.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:13:09.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:13:09.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:13:09.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:13:09.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:13:09.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:13:09.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:13:09.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:13:10.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-05-20T06:11:27.170489+0000 '>' 2024-06-03T06:12:41.705047+0000 2024-06-03T06:13:10.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:13:11.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:13:11.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:13:11.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:13:11.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:13:11.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:13:11.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:13:11.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:13:11.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-05-20T06:11:27.170489+0000 '>' 2024-06-03T06:12:41.705047+0000 2024-06-03T06:13:11.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:13:12.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:13:12.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:13:12.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:13:12.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:13:12.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:13:12.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:13:12.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:13:12.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-05-20T06:11:27.170489+0000 '>' 2024-06-03T06:12:41.705047+0000 2024-06-03T06:13:12.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:13:13.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:13:13.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:13:13.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:13:13.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:13:13.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:13:13.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:13:13.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:13:13.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-05-20T06:11:27.170489+0000 '>' 2024-06-03T06:12:41.705047+0000 2024-06-03T06:13:13.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:13:14.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:13:14.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:13:14.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:13:14.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:13:14.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:13:14.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:13:14.953 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:13:15.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:13:07.917451+0000 '>' 2024-06-03T06:12:41.705047+0000 2024-06-03T06:13:15.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:13:15.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-06-03T06:13:15.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:13:15.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:13:15.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:13:15.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:13:15.253 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:13:15.253 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:13:15.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:13:15.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:13:15.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:13:15.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:13:15.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:13:15.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:13:15.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:13:15.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:13:15.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:13:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:13:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:13:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:13:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:13:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:13:15.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:13:15.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215109 2024-06-03T06:13:15.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215109 2024-06-03T06:13:15.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109' 2024-06-03T06:13:15.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:13:15.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:13:15.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-06-03T06:13:15.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-06-03T06:13:15.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109 1-42949672983' 2024-06-03T06:13:15.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:13:15.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:13:15.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-06-03T06:13:15.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-06-03T06:13:15.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109 1-42949672983 2-64424509451' 2024-06-03T06:13:15.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:13:15.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215109 2024-06-03T06:13:15.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:13:15.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:13:15.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215109 2024-06-03T06:13:15.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:13:15.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215109 2024-06-03T06:13:15.965 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 103079215109 2024-06-03T06:13:15.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215109' 2024-06-03T06:13:15.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:13:16.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215109 -lt 103079215109 2024-06-03T06:13:16.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:13:16.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-06-03T06:13:16.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:13:16.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:13:16.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-06-03T06:13:16.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:13:16.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-06-03T06:13:16.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-06-03T06:13:16.266 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672983 2024-06-03T06:13:16.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:13:16.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-06-03T06:13:16.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:13:16.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-06-03T06:13:16.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:13:16.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:13:16.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-06-03T06:13:16.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:13:16.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-06-03T06:13:16.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-06-03T06:13:16.561 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509451 2024-06-03T06:13:16.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:13:16.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-06-03T06:13:16.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:13:16.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:13:16.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:13:17.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:13:17.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:13:17.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:13:17.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:13:17.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:13:17.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:13:17.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:13:17.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:13:17.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:13:17.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:13:17.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:13:17.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:13:17.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:13:17.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:13:17.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:13:17.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-06-03T06:13:18.130 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:13:18.130 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-03T06:13:08.155531+0000 20'1 25:57 [1,0] 1 [1,0] 1 20'1 2024-06-03T06:13:07.917451+0000 20'1 2024-06-03T06:13:07.917451+0000 0 1 periodic scrub scheduled @ 2024-06-04T06:13:07.917451+0000 1 0 2024-06-03T06:13:18.130 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:13:18.130 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:13:18.130 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:13:18.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-06-03T06:13:18.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-03T06:13:18.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:13:18.143 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-03T06:13:18.143 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:13:18.143 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:13:18.143 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:13:18.143 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:13:18.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T06:13:18.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:13:18.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T06:13:18.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-03T06:13:18.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:13:18.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:13:18.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:13:18.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:13:18.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-03T06:13:18.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:13:18.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:13:18.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:13:18.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:13:18.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:13:18.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:13:18.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:13:18.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:13:18.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:13:18.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:13:19.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:13:19.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-03T06:13:19.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:13:19.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:13:19.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:13:19.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:13:19.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:13:19.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-06-03T06:13:20.308 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:13:20.308 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:13:20.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T06:13:20.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:13:20.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:13:20.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:13:20.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:13:20.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:13:20.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:13:20.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:13:20.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:13:20.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:13:20.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:13:20.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:13:20.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:13:20.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:13:20.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:13:20.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:13:20.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:13:20.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:13:20.584 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:13:20.584 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:13:20.584 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:13:20.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:13:20.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:13:20.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:13:20.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:13:20.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:13:20.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:13:20.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:13:20.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:13:20.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:13:20.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:13:20.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:13:20.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:13:20.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:13:20.586 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:13:20.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:13:20.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:13:20.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:13:20.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:13:20.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:13:20.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:13:20.615 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:20.615+0000 7f471aa13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:13:20.617 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:20.617+0000 7f471aa13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:13:20.618 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:20.618+0000 7f471aa13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:13:20.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:13:20.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:13:20.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:13:20.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:13:20.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:13:20.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:20.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:13:20.888 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:13:20.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:20.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:21.133 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:21.133+0000 7f471aa13640 -1 Falling back to public interface 2024-06-03T06:13:21.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:22.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:22.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:22.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:13:22.182 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:13:22.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:22.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:22.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:23.465 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:23.465+0000 7f471aa13640 -1 osd.0 25 log_to_monitors true 2024-06-03T06:13:23.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:23.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:13:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:13:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:24.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:24.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:24.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:13:24.797 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:13:24.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:24.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:25.112 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/2023585247,v1:127.0.0.1:6803/2023585247] [v2:127.0.0.1:6804/2023585247,v1:127.0.0.1:6805/2023585247] exists,up 7220952b-1537-4720-8cac-2770e07493b3 2024-06-03T06:13:25.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:13:25.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:13:25.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:13:25.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:13:25.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:13:25.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:13:25.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:13:25.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:13:25.112 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:13:25.113 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:13:25.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:13:25.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:13:25.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:13:25.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:13:25.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:13:25.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:13:25.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:13:25.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:13:25.205 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:13:25.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:13:25.493 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:13:25.493 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:13:25.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:13:25.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:13:25.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:13:25.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-06-03T06:13:25.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-06-03T06:13:25.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-06-03T06:13:25.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:13:25.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:13:25.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-06-03T06:13:25.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-06-03T06:13:25.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672986' 2024-06-03T06:13:25.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:13:25.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:13:25.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-06-03T06:13:25.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-06-03T06:13:25.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672986 2-64424509454' 2024-06-03T06:13:25.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:13:25.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-06-03T06:13:25.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:13:25.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:13:25.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-06-03T06:13:25.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:13:25.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-06-03T06:13:25.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-06-03T06:13:25.820 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 124554051586 2024-06-03T06:13:25.820 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:13:26.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-06-03T06:13:26.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:13:27.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:13:27.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:13:27.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-06-03T06:13:27.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:13:27.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-06-03T06:13:27.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:13:27.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:13:27.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-06-03T06:13:27.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:13:27.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-06-03T06:13:27.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-06-03T06:13:27.422 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672986 2024-06-03T06:13:27.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:13:27.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672986 2024-06-03T06:13:27.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:13:27.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-06-03T06:13:27.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:13:27.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:13:27.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-06-03T06:13:27.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:13:27.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-06-03T06:13:27.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-06-03T06:13:27.715 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509454 2024-06-03T06:13:27.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:13:28.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-06-03T06:13:28.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:13:28.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:13:28.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:13:28.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:13:28.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:13:28.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:13:28.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:13:28.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:13:28.363 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:13:28.363 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:13:28.363 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:13:28.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:13:28.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:13:28.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:13:28.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:13:29.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:13:29.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:13:29.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:13:29.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-06-03T06:13:29.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-03T06:13:29.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:13:29.007 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-03T06:13:29.007 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:13:29.007 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:13:29.007 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:13:29.007 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:13:29.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T06:13:29.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:13:29.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T06:13:29.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-03T06:13:29.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:13:29.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:13:29.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:13:29.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:13:29.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-03T06:13:29.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:13:29.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:13:29.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:13:29.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:13:29.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:13:29.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:13:29.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:13:29.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:13:29.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:13:29.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:13:29.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:13:29.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-03T06:13:29.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:13:29.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:13:29.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:13:29.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:13:29.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:13:29.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-03T06:13:31.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T06:13:31.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:13:31.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:13:31.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:13:31.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:13:31.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:13:31.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:13:31.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:13:31.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:13:31.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:13:31.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:13:31.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:13:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:13:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:13:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:13:31.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:13:31.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:13:31.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:13:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:13:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:13:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:13:31.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:13:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:13:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:13:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:13:31.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:13:31.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:13:31.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:13:31.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:13:31.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:13:31.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:13:31.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:13:31.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:13:31.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:13:31.513 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:13:31.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:13:31.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:13:31.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:13:31.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:13:31.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:13:31.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:13:31.542 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:31.542+0000 7f2e689da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:13:31.544 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:31.545+0000 7f2e689da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:13:31.546 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:31.546+0000 7f2e689da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:13:31.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:13:31.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:13:31.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:13:31.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:13:31.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:13:31.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:31.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:13:31.815 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:13:31.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:31.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:32.061 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:32.061+0000 7f2e689da640 -1 Falling back to public interface 2024-06-03T06:13:32.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:33.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:33.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:33.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:13:33.110 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:13:33.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:33.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:33.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:34.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:34.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:34.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:13:34.411 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:13:34.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:34.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:34.503 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:34.503+0000 7f2e689da640 -1 osd.0 30 log_to_monitors true 2024-06-03T06:13:34.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:35.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:35.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:35.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:13:35.769 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:13:35.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:35.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:36.140 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 34 up_thru 0 down_at 31 last_clean_interval [29,30) [v2:127.0.0.1:6802/1524121808,v1:127.0.0.1:6803/1524121808] [v2:127.0.0.1:6804/1524121808,v1:127.0.0.1:6805/1524121808] exists,up 7220952b-1537-4720-8cac-2770e07493b3 2024-06-03T06:13:36.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:13:36.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:13:36.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:13:36.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:13:36.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:13:36.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:13:36.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:13:36.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:13:36.141 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:13:36.141 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:13:36.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:13:36.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:13:36.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:13:36.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:13:36.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:13:36.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:13:36.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:13:36.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:13:36.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:13:36.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:13:36.520 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:13:36.520 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:13:36.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:13:36.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:13:36.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:13:36.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-06-03T06:13:36.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-06-03T06:13:36.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066' 2024-06-03T06:13:36.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:13:36.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:13:36.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672989 2024-06-03T06:13:36.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672989 2024-06-03T06:13:36.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-42949672989' 2024-06-03T06:13:36.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:13:36.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:13:36.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-06-03T06:13:36.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-06-03T06:13:36.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-42949672989 2-64424509458' 2024-06-03T06:13:36.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:13:36.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888066 2024-06-03T06:13:36.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:13:36.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:13:36.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888066 2024-06-03T06:13:36.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:13:36.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-06-03T06:13:36.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-06-03T06:13:36.846 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 146028888066 2024-06-03T06:13:36.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:13:37.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 146028888066 2024-06-03T06:13:37.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:13:38.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:13:38.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:13:38.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888066 -lt 146028888066 2024-06-03T06:13:38.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:13:38.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672989 2024-06-03T06:13:38.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:13:38.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:13:38.438 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672989 2024-06-03T06:13:38.438 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:13:38.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672989 2024-06-03T06:13:38.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-06-03T06:13:38.439 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672989 2024-06-03T06:13:38.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:13:38.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672990 -lt 42949672989 2024-06-03T06:13:38.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:13:38.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-06-03T06:13:38.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:13:38.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:13:38.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-06-03T06:13:38.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:13:38.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-06-03T06:13:38.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-06-03T06:13:38.728 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509458 2024-06-03T06:13:38.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:13:39.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-06-03T06:13:39.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:13:39.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:13:39.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:13:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:13:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:13:39.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:13:39.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:13:39.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:13:39.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:13:39.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:13:39.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:13:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:13:39.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:13:39.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:13:39.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:13:40.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:13:40.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:13:40.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:13:40.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:13:40.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-03T06:13:40.049 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:13:07.916+0000 7fb5cd6e7640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-03T06:13:40.049 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:13:07.916+0000 7fb5cd6e7640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-03T06:13:40.049 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:13:07.916+0000 7fb5cd6e7640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-06-03T06:13:40.049 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:13:07.916+0000 7fb5cd6e7640 20 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-06-03T06:13:40.049 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:13:07.916+0000 7fb5cd6e7640 19 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-03T06:13:40.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T06:13:40.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:13:40.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:13:40.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:13:40.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:13:40.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:13:40.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:13:40.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:13:40.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:13:40.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:13:40.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:13:40.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:13:40.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:13:40.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:13:40.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:13:40.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:13:40.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:13:40.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:13:40.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:13:40.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:13:40.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:13:40.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:13:40.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:13:40.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:13:40.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:13:40.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:13:40.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:13:40.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:13:40.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:13:40.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T06:13:40.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T06:13:40.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T06:13:40.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T06:13:40.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:13:40.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:13:40.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:13:40.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:13:40.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:13:40.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:13:40.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:13:40.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:13:40.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:13:40.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:13:40.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:13:40.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:13:40.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:13:40.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:13:40.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:13:40.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:13:40.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:13:40.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:13:40.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:13:40.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:13:40.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:13:40.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:13:40.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:13:40.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:13:40.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:13:40.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:13:40.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:13:40.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:13:40.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T06:13:40.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:13:40.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:13:40.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:13:40.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T06:13:40.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:13:40.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T06:13:40.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:13:40.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T06:13:40.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-06-03T06:13:40.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-06-03T06:13:40.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-06-03T06:13:40.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-06-03T06:13:40.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T06:13:40.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T06:13:40.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T06:13:40.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T06:13:40.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T06:13:40.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T06:13:40.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T06:13:40.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:13:40.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:13:40.576 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:13:40.576 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:13:40.576 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:13:40.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:13:40.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T06:13:40.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T06:13:40.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T06:13:40.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:13:40.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:13:40.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T06:13:40.770 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T06:13:40.770 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:13:40.771 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:13:40.771 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:13:40.771 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:13:40.771 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:13:40.771 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:13:40.771 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:13:40.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:13:40.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T06:13:40.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T06:13:40.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:13:40.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:13:40.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T06:13:40.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T06:13:40.857 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:13:40.857 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:13:40.857 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:13:40.857 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:13:40.857 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:13:40.858 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:13:40.858 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:13:40.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:13:40.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T06:13:40.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-06-03T06:13:40.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T06:13:40.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T06:13:40.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T06:13:40.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T06:13:40.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T06:13:40.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T06:13:41.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T06:13:41.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:13:41.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:13:41.755 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:13:41.755 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:13:41.755 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:13:41.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:13:41.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:13:41.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:13:41.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:13:41.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-06-03T06:13:41.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-06-03T06:13:41.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:13:41.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:13:41.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:13:41.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T06:13:41.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:13:41.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:13:41.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:13:41.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:13:41.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:13:41.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:13:41.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:13:41.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:13:41.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:13:41.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:13:41.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:13:41.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:13:41.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:13:41.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:13:41.791 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:13:41.791 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:13:41.791 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:13:41.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:13:41.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:13:41.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:13:41.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:13:41.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:13:41.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:13:41.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:13:41.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:13:41.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:13:41.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:13:41.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:13:41.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:13:41.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:13:41.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:13:41.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:13:41.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f03eb923-99a6-4d3a-bfae-ada3746cb600 2024-06-03T06:13:41.792 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 f03eb923-99a6-4d3a-bfae-ada3746cb600 2024-06-03T06:13:41.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f03eb923-99a6-4d3a-bfae-ada3746cb600' 2024-06-03T06:13:41.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:13:41.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAVX11mkdg5MBAA22EiTwpZTiEhRoKEJfA+Pw== 2024-06-03T06:13:41.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAVX11mkdg5MBAA22EiTwpZTiEhRoKEJfA+Pw=="}' 2024-06-03T06:13:41.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f03eb923-99a6-4d3a-bfae-ada3746cb600 -i td/osd-scrub-repair/0/new.json 2024-06-03T06:13:42.068 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:13:42.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T06:13:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAVX11mkdg5MBAA22EiTwpZTiEhRoKEJfA+Pw== --osd-uuid f03eb923-99a6-4d3a-bfae-ada3746cb600 2024-06-03T06:13:42.110 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:42.110+0000 7f5b4c60d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:13:42.119 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:42.119+0000 7f5b4c60d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:13:42.120 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:42.120+0000 7f5b4c60d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:13:42.120 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:42.121+0000 7f5b4c60d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:13:42.121 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:42.121+0000 7f5b4c60d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:13:42.121 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:42.121+0000 7f5b4c60d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:13:42.211 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:42.211+0000 7f5b4c60d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T06:13:47.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T06:13:47.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:13:47.759 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:13:47.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T06:13:47.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:13:48.143 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:13:48.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T06:13:48.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:13:48.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:13:48.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:13:48.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:13:48.173 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:48.172+0000 7fec8d185640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:13:48.201 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:48.199+0000 7fec8d185640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:13:48.202 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:48.200+0000 7fec8d185640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:13:48.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T06:13:48.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:13:48.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:13:48.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:13:48.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:13:48.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:48.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:13:48.476 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:13:48.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:48.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:48.737 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:48.736+0000 7fec8d185640 -1 Falling back to public interface 2024-06-03T06:13:48.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:49.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:49.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:49.764 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:13:49.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:13:49.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:49.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:50.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:51.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:51.064 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:13:51.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:51.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:13:51.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:51.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:51.067 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:13:51.067+0000 7fec8d185640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:13:51.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:52.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:52.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:52.495 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:13:52.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:13:52.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:52.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:52.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:53.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:53.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:53.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:13:53.804 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:13:53.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:53.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:54.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:55.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:55.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:55.106 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:13:55.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:13:55.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:55.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:55.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:56.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:56.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:56.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:13:56.407 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:13:56.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:56.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:56.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:57.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:57.799 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:13:57.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:57.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:13:57.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:57.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:58.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:13:59.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:13:59.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:13:59.100 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:13:59.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:13:59.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:13:59.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:13:59.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:00.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:00.402 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:14:00.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:00.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:14:00.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:00.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:14:00.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:01.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:01.798 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:14:01.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:01.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:14:01.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:01.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:14:02.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:03.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:03.098 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:14:03.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:03.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:14:03.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:03.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:14:03.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:04.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:14:04.763 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:14:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:14:05.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:06.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:06.198 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:14:06.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:06.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:14:06.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:06.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:14:06.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:07.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:07.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:07.893 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:14:07.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:14:07.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:07.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:14:08.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:09.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:09.270 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:14:09.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:09.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:14:09.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:09.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:14:09.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:10.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:10.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:10.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:14:10.814 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:14:10.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:10.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:14:11.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:12.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:12.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:12.393 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:14:12.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:14:12.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:12.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:14:12.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:13.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:13.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:13.862 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:14:13.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:14:13.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:13.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:14:14.164 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3700553625,v1:127.0.0.1:6803/3700553625] [v2:127.0.0.1:6804/3700553625,v1:127.0.0.1:6805/3700553625] exists,up f03eb923-99a6-4d3a-bfae-ada3746cb600 2024-06-03T06:14:14.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:14:14.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:14:14.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:14:14.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-06-03T06:14:14.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:14:14.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:14:14.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:14:14.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T06:14:14.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:14:14.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:14:14.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:14:14.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:14:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:14:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:14:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:14:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:14:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:14:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:14:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:14:14.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:14:14.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:14:14.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:14:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:14:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:14:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:14:14.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:14:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:14:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:14:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:14:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:14:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:14:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:14:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:14:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:14:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:14:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:14:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:14:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:14:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:14:14.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:14:14.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4ce04533-c54d-430d-b5eb-85292391a87d 2024-06-03T06:14:14.171 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 4ce04533-c54d-430d-b5eb-85292391a87d 2024-06-03T06:14:14.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4ce04533-c54d-430d-b5eb-85292391a87d' 2024-06-03T06:14:14.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:14:14.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA2X11mNq4oCxAAVgQopeHDgMVIrJW8plPrmQ== 2024-06-03T06:14:14.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA2X11mNq4oCxAAVgQopeHDgMVIrJW8plPrmQ=="}' 2024-06-03T06:14:14.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4ce04533-c54d-430d-b5eb-85292391a87d -i td/osd-scrub-repair/1/new.json 2024-06-03T06:14:14.562 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:14:14.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T06:14:14.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA2X11mNq4oCxAAVgQopeHDgMVIrJW8plPrmQ== --osd-uuid 4ce04533-c54d-430d-b5eb-85292391a87d 2024-06-03T06:14:14.605 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:14.605+0000 7efe10a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:14:14.647 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:14.647+0000 7efe10a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:14:14.648 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:14.648+0000 7efe10a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:14:14.648 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:14.648+0000 7efe10a15640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:14:14.648 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:14.648+0000 7efe10a15640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:14:14.648 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:14.648+0000 7efe10a15640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:14:14.724 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:14.724+0000 7efe10a15640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T06:14:20.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T06:14:20.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:14:20.255 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:14:20.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T06:14:20.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:14:20.628 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:14:20.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T06:14:20.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:14:20.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:14:20.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:14:20.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:14:20.658 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:20.658+0000 7fa75c024640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:14:20.660 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:20.660+0000 7fa75c024640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:14:20.661 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:20.661+0000 7fa75c024640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:14:20.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T06:14:20.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:14:20.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:14:20.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:14:20.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:14:20.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:20.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:14:20.917 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:14:20.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:20.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:21.180 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:21.180+0000 7fa75c024640 -1 Falling back to public interface 2024-06-03T06:14:21.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:22.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:22.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:22.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:14:22.205 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:14:22.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:22.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:22.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:23.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:23.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:23.507 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:14:23.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:14:23.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:23.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:23.512 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:23.512+0000 7fa75c024640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:14:23.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:24.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:24.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:24.879 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:14:24.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:14:24.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:24.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:25.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:26.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:26.244 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:14:26.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:26.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:14:26.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:26.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:26.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:27.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:27.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:27.545 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:14:27.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:14:27.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:27.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:27.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:28.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:28.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:28.845 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:14:28.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:14:28.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:28.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:29.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:30.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:30.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:30.147 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:14:30.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:14:30.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:30.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:30.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:31.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:31.558 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:14:31.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:31.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:14:31.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:31.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:31.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:32.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:32.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:32.860 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:14:32.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:14:32.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:32.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:33.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:34.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:34.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:34.169 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:14:34.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:14:34.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:34.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:34.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:35.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:35.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:35.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:14:35.548 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:14:35.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:35.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:35.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:36.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:36.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:36.848 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:14:36.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:14:36.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:36.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:37.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:38.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:38.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:38.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:14:38.426 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:14:38.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:38.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:38.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:39.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:39.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:39.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:14:39.759 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:14:39.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:39.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:40.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:41.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:41.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:41.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:14:41.153 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:14:41.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:41.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:41.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:42.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:42.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:42.889 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:14:42.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:14:42.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:42.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:43.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:44.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:44.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:44.360 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:14:44.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:14:44.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:44.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:44.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:45.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:45.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:45.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:14:45.693 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:14:45.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:45.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:46.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:47.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:47.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:47.034 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:14:47.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:14:47.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:47.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:14:47.330 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3965343573,v1:127.0.0.1:6811/3965343573] [v2:127.0.0.1:6812/3965343573,v1:127.0.0.1:6813/3965343573] exists,up 4ce04533-c54d-430d-b5eb-85292391a87d 2024-06-03T06:14:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:14:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:14:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:14:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-06-03T06:14:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:14:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:14:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:14:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T06:14:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:14:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:14:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:14:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:14:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:14:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:14:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:14:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:14:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:14:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:14:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:14:47.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:14:47.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:14:47.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:14:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:14:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:14:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:14:47.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:14:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:14:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:14:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:14:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:14:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:14:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:14:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:14:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:14:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:14:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:14:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:14:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:14:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:14:47.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:14:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1817356c-1871-4cdc-bd84-b28ddf872b8e 2024-06-03T06:14:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1817356c-1871-4cdc-bd84-b28ddf872b8e' 2024-06-03T06:14:47.338 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 1817356c-1871-4cdc-bd84-b28ddf872b8e 2024-06-03T06:14:47.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:14:47.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXX11mTTIIFRAAq3HuOwAigC799lzjw7zptg== 2024-06-03T06:14:47.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXX11mTTIIFRAAq3HuOwAigC799lzjw7zptg=="}' 2024-06-03T06:14:47.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1817356c-1871-4cdc-bd84-b28ddf872b8e -i td/osd-scrub-repair/2/new.json 2024-06-03T06:14:47.739 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:14:47.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-03T06:14:47.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBXX11mTTIIFRAAq3HuOwAigC799lzjw7zptg== --osd-uuid 1817356c-1871-4cdc-bd84-b28ddf872b8e 2024-06-03T06:14:47.783 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:47.782+0000 7f37961a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:14:47.823 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:47.823+0000 7f37961a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:14:47.824 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:47.824+0000 7f37961a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:14:47.825 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:47.825+0000 7f37961a3640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:14:47.825 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:47.825+0000 7f37961a3640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:14:47.825 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:47.825+0000 7f37961a3640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:14:47.890 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:47.890+0000 7f37961a3640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T06:14:54.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-03T06:14:54.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:14:54.206 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T06:14:54.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T06:14:54.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:14:54.582 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:14:54.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T06:14:54.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:14:54.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:14:54.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:14:54.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:14:54.613 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:54.613+0000 7f0e53a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:14:54.615 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:54.615+0000 7f0e53a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:14:54.616 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:54.617+0000 7f0e53a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:14:54.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T06:14:54.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:14:54.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:14:54.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:14:54.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:14:54.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:54.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:14:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:14:54.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:54.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:14:55.133 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:55.133+0000 7f0e53a11640 -1 Falling back to public interface 2024-06-03T06:14:55.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:56.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:56.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:56.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:14:56.179 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:14:56.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:56.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:14:56.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:57.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:57.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:57.481 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:14:57.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:14:57.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:57.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:14:57.523 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:14:57.523+0000 7f0e53a11640 -1 osd.2 0 log_to_monitors true 2024-06-03T06:14:57.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:14:58.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:14:58.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:14:58.832 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:14:58.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:14:58.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:14:58.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:14:59.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:00.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:00.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:00.133 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:15:00.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:15:00.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:00.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:00.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:01.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:01.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:01.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:15:01.434 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:15:01.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:01.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:01.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:02.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:02.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:02.736 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:15:02.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:15:02.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:02.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:03.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:04.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:04.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:04.092 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:15:04.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:15:04.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:04.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:04.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:05.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:05.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:05.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:15:05.398 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:15:05.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:05.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:05.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:06.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:06.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:06.727 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:15:06.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:15:06.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:06.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:07.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:08.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:08.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:08.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:15:08.114 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:15:08.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:08.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:08.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:09.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:09.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:09.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:15:09.555 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:15:09.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:09.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:09.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:10.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:10.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:10.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:15:10.858 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:15:10.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:10.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:11.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:12.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:12.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:12.469 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:15:12.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:15:12.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:12.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:12.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:13.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:13.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:13.820 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:15:13.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:15:13.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:13.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:14.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:15.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:15.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:15.230 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:15:15.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:15:15.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:15.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:15.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:16.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:16.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:16.806 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:15:16.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:15:16.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:16.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:17.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:18.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:18.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:18.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:15:18.111 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:15:18.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:18.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:18.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:19.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:19.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:19.753 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:15:19.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:15:19.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:19.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:20.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:21.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:21.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:21.056 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:15:21.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:15:21.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:21.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:15:21.368 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2332196705,v1:127.0.0.1:6819/2332196705] [v2:127.0.0.1:6820/2332196705,v1:127.0.0.1:6821/2332196705] exists,up 1817356c-1871-4cdc-bd84-b28ddf872b8e 2024-06-03T06:15:21.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:15:21.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:15:21.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:15:21.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-06-03T06:15:21.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-06-03T06:15:21.750 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'testpool' created 2024-06-03T06:15:21.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:15:22.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-06-03T06:15:23.255 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 2 2024-06-03T06:15:23.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-06-03T06:15:23.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:15:23.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:15:23.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:15:23.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:15:23.268 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:15:23.268 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:15:23.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:15:23.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:15:23.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:15:23.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:15:23.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:15:23.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:15:23.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:15:23.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:15:23.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:15:23.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:15:23.811 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:15:23.811 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:15:23.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:15:23.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:15:23.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:15:23.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-06-03T06:15:23.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-06-03T06:15:23.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-06-03T06:15:23.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:15:23.953 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:15:24.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-06-03T06:15:24.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-06-03T06:15:24.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672974' 2024-06-03T06:15:24.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:15:24.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:15:24.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-06-03T06:15:24.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-06-03T06:15:24.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672974 2-64424509442' 2024-06-03T06:15:24.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:15:24.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-06-03T06:15:24.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:15:24.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:15:24.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-06-03T06:15:24.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:15:24.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-06-03T06:15:24.218 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836498 2024-06-03T06:15:24.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-06-03T06:15:24.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:15:24.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-06-03T06:15:24.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:15:25.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:15:25.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:15:25.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-06-03T06:15:25.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:15:26.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T06:15:26.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:15:27.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-06-03T06:15:27.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:15:27.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-06-03T06:15:27.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:15:27.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:15:27.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-06-03T06:15:27.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:15:27.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-06-03T06:15:27.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-06-03T06:15:27.114 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672974 2024-06-03T06:15:27.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:15:27.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-06-03T06:15:27.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:15:27.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-06-03T06:15:27.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:15:27.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:15:27.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-06-03T06:15:27.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:15:27.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-06-03T06:15:27.412 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509442 2024-06-03T06:15:27.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-06-03T06:15:27.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:15:27.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-06-03T06:15:27.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:15:27.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:15:27.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:15:28.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:15:28.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:15:28.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:15:28.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:15:28.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:15:28.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:15:28.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:15:28.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:15:28.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:15:28.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:15:28.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:15:28.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:15:28.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:15:28.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:15:28.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:15:28.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-06-03T06:15:28.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-06-03T06:15:28.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-06-03T06:15:28.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-03T06:15:28.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:15:28.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-03T06:15:28.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:15:29.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-03T06:15:29.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:15:29.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-03T06:15:29.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:15:29.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-03T06:15:29.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:15:29.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-03T06:15:29.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:15:29.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-03T06:15:29.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:15:29.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-03T06:15:29.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:15:29.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-03T06:15:29.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:15:30.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-03T06:15:30.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:15:30.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-06-03T06:15:30.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-03T06:15:30.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:15:30.146 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-03T06:15:30.146 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:15:30.146 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:15:30.146 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:15:30.146 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:15:30.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T06:15:30.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:15:30.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T06:15:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-06-03T06:15:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:15:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:15:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:15:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:15:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-06-03T06:15:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:15:30.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:15:30.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:15:30.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:15:30.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:15:30.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:15:30.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:15:30.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:15:30.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:15:30.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:15:31.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:15:31.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-06-03T06:15:31.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:15:31.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:15:31.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:15:31.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:15:31.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:15:31.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-06-03T06:15:32.364 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:ff7b1f36:::obj1:head# 2024-06-03T06:15:32.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:15:32.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:15:32.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:15:32.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:15:32.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:15:32.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:15:32.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:15:32.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:15:32.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:15:32.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:15:32.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:15:32.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:15:32.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:15:32.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:15:32.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:15:32.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:15:32.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:15:32.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:15:32.884 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:15:32.884 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:15:32.884 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:15:32.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:15:32.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:15:32.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:15:32.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:15:32.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:15:32.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:15:32.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:15:32.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:15:32.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:15:32.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:15:32.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:15:32.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:15:32.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:15:32.885 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:15:32.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:15:32.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:15:32.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:15:32.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:15:32.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:15:32.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:15:32.915 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:15:32.914+0000 7f199a399640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:15:32.955 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:15:32.954+0000 7f199a399640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:15:32.956 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:15:32.955+0000 7f199a399640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:15:33.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:15:33.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:15:33.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:15:33.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:15:33.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:15:33.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:33.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:15:33.185 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:15:33.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:33.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:15:33.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:33.478 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:15:33.477+0000 7f199a399640 -1 Falling back to public interface 2024-06-03T06:15:34.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:34.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:34.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:15:34.474 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:15:34.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:34.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:15:34.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:35.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:35.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:35.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:15:35.775 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:15:35.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:35.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:15:35.850 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:15:35.849+0000 7f199a399640 -1 osd.0 21 log_to_monitors true 2024-06-03T06:15:36.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:37.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:37.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:37.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:15:37.082 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:15:37.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:37.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:15:37.379 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/2640467187,v1:127.0.0.1:6803/2640467187] [v2:127.0.0.1:6804/2640467187,v1:127.0.0.1:6805/2640467187] exists,up f03eb923-99a6-4d3a-bfae-ada3746cb600 2024-06-03T06:15:37.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:15:37.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:15:37.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:15:37.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:15:37.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:15:37.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:15:37.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:15:37.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:15:37.380 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:15:37.380 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:15:37.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:15:37.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:15:37.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:15:37.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:15:37.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:15:37.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:15:37.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:15:37.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:15:37.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:15:37.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:15:37.765 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:15:37.765 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:15:37.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:15:37.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:15:37.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:15:37.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-06-03T06:15:37.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-06-03T06:15:37.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-06-03T06:15:37.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:15:37.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:15:37.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-06-03T06:15:37.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-06-03T06:15:37.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978' 2024-06-03T06:15:37.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:15:37.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:15:38.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-06-03T06:15:38.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-06-03T06:15:38.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978 2-64424509446' 2024-06-03T06:15:38.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:15:38.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-06-03T06:15:38.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:15:38.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:15:38.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-06-03T06:15:38.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:15:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-06-03T06:15:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-06-03T06:15:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 107374182402 2024-06-03T06:15:38.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:15:38.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-06-03T06:15:38.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:15:38.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-06-03T06:15:38.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:15:38.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:15:38.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-06-03T06:15:38.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:15:38.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-06-03T06:15:38.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-06-03T06:15:38.383 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672978 2024-06-03T06:15:38.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:15:38.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672978 2024-06-03T06:15:38.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:15:39.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:15:39.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:15:39.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672978 2024-06-03T06:15:39.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:15:40.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T06:15:40.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:15:41.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-06-03T06:15:41.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:15:41.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-06-03T06:15:41.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:15:41.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:15:41.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-06-03T06:15:41.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:15:41.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-06-03T06:15:41.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-06-03T06:15:41.285 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509446 2024-06-03T06:15:41.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:15:41.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-06-03T06:15:41.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:15:41.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:15:41.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:15:41.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:15:41.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:15:41.943 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:15:41.943 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:15:41.943 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:15:41.943 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:15:41.944 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:15:41.944 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:15:42.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:15:42.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:15:42.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:15:42.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:15:42.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:15:42.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:15:42.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:15:42.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-06-03T06:15:42.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-03T06:15:42.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:15:42.598 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-03T06:15:42.598 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:15:42.598 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:15:42.598 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:15:42.599 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:15:42.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T06:15:42.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:15:42.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T06:15:43.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-06-03T06:15:43.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:15:43.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:15:43.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:15:43.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:15:43.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-06-03T06:15:43.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:15:43.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:15:43.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:15:43.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:15:43.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:15:43.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:15:43.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:15:43.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:15:43.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:15:43.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:15:43.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:15:43.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-06-03T06:15:43.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:15:43.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:15:43.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:15:43.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:15:43.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:15:43.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-06-03T06:15:44.700 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:104778fc:::obj2:head# 2024-06-03T06:15:45.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:15:45.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:15:45.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:15:45.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:15:45.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:15:45.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:15:45.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:15:45.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:15:45.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:15:45.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:15:45.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:15:45.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:15:45.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:15:45.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:15:45.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:15:45.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:15:45.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:15:45.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:15:45.229 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:15:45.229 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:15:45.230 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:15:45.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:15:45.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:15:45.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:15:45.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:15:45.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:15:45.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:15:45.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:15:45.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:15:45.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:15:45.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:15:45.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:15:45.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:15:45.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:15:45.231 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:15:45.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:15:45.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:15:45.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:15:45.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:15:45.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:15:45.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:15:45.260 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:15:45.260+0000 7f611ecb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:15:45.262 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:15:45.263+0000 7f611ecb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:15:45.264 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:15:45.264+0000 7f611ecb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:15:45.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:15:45.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:15:45.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:15:45.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:15:45.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:15:45.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:45.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:15:45.531 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:15:45.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:45.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:15:45.784 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:15:45.784+0000 7f611ecb1640 -1 Falling back to public interface 2024-06-03T06:15:45.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:46.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:46.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:46.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:15:46.820 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:15:46.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:46.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:15:47.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:48.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:48.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:48.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:15:48.157 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:15:48.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:48.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:15:48.198 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:15:48.198+0000 7f611ecb1640 -1 osd.0 26 log_to_monitors true 2024-06-03T06:15:48.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:49.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:49.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:49.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:15:49.459 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:15:49.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:49.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:15:49.802 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/3211262747,v1:127.0.0.1:6803/3211262747] [v2:127.0.0.1:6804/3211262747,v1:127.0.0.1:6805/3211262747] exists,up f03eb923-99a6-4d3a-bfae-ada3746cb600 2024-06-03T06:15:49.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:15:49.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:15:49.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:15:49.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:15:49.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:15:49.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:15:49.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:15:49.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:15:49.803 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:15:49.803 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:15:49.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:15:49.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:15:49.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:15:49.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:15:49.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:15:49.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:15:49.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:15:49.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:15:49.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:15:50.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:15:50.260 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:15:50.260 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:15:50.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:15:50.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:15:50.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:15:50.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-06-03T06:15:50.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-06-03T06:15:50.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-06-03T06:15:50.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:15:50.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:15:50.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-06-03T06:15:50.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-06-03T06:15:50.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672981' 2024-06-03T06:15:50.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:15:50.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:15:50.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-06-03T06:15:50.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-06-03T06:15:50.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672981 2-64424509449' 2024-06-03T06:15:50.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:15:50.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-06-03T06:15:50.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:15:50.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:15:50.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-06-03T06:15:50.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:15:50.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-06-03T06:15:50.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-06-03T06:15:50.588 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 128849018882 2024-06-03T06:15:50.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:15:50.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-06-03T06:15:50.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:15:51.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:15:51.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:15:52.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-06-03T06:15:52.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:15:52.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-06-03T06:15:52.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:15:52.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:15:52.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-06-03T06:15:52.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:15:52.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-06-03T06:15:52.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-06-03T06:15:52.201 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672981 2024-06-03T06:15:52.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:15:52.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-06-03T06:15:52.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:15:52.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-06-03T06:15:52.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:15:52.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:15:52.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-06-03T06:15:52.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:15:52.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-06-03T06:15:52.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-06-03T06:15:52.493 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509449 2024-06-03T06:15:52.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:15:52.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509449 2024-06-03T06:15:52.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:15:52.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:15:52.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:15:53.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:15:53.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:15:53.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:15:53.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:15:53.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:15:53.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:15:53.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:15:53.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:15:53.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:15:53.422 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:15:53.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:15:53.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:15:53.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:15:53.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:15:53.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:15:53.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-06-03T06:15:53.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:15:53.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:15:53.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:15:53.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:15:54.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-03T06:15:54.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:15:54.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:15:54.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:15:54.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:15:54.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-03T06:15:54.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:15:54.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:15:54.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:15:54.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:15:54.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:15:54.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:15:54.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:15:54.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:15:54.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:15:54.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:15:54.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:15:54.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-03T06:15:54.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:15:54.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:15:54.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:15:54.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:15:54.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:15:54.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-06-03T06:15:56.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:15:56.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:15:56.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:15:56.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:15:56.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:15:56.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:15:56.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:15:56.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:15:56.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:15:56.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:15:56.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:15:56.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:15:56.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:15:56.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:15:56.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:15:56.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:15:56.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:15:56.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:15:56.242 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:15:56.242 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:15:56.242 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:15:56.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:15:56.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:15:56.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:15:56.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:15:56.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:15:56.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:15:56.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:15:56.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:15:56.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:15:56.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:15:56.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:15:56.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:15:56.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:15:56.244 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:15:56.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:15:56.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:15:56.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:15:56.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:15:56.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:15:56.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:15:56.275 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:15:56.275+0000 7f3acce11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:15:56.277 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:15:56.277+0000 7f3acce11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:15:56.279 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:15:56.279+0000 7f3acce11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:15:56.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:15:56.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:15:56.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:15:56.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:15:56.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:15:56.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:56.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:15:56.549 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:15:56.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:56.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:15:56.793 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:15:56.793+0000 7f3acce11640 -1 Falling back to public interface 2024-06-03T06:15:56.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:57.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:57.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:57.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:15:57.839 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:15:57.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:57.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:15:58.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:15:59.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:15:59.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:15:59.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:15:59.145 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:15:59.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:15:59.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:15:59.252 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:15:59.252+0000 7f3acce11640 -1 osd.1 31 log_to_monitors true 2024-06-03T06:15:59.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:16:00.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:16:00.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:16:00.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:16:00.452 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:16:00.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:16:00.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:16:00.756 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/1331989475,v1:127.0.0.1:6811/1331989475] [v2:127.0.0.1:6812/1331989475,v1:127.0.0.1:6813/1331989475] exists,up 4ce04533-c54d-430d-b5eb-85292391a87d 2024-06-03T06:16:00.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:16:00.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:16:00.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:16:00.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:16:00.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:16:00.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:16:00.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:16:00.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:16:00.758 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:16:00.758 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:16:00.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:16:00.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:16:00.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:16:00.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:16:00.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:16:00.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:16:00.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:16:00.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:16:00.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:16:01.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:16:01.138 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:16:01.138 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:16:01.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:16:01.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:01.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:16:01.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-06-03T06:16:01.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-06-03T06:16:01.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885' 2024-06-03T06:16:01.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:01.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:16:01.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-06-03T06:16:01.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-06-03T06:16:01.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362' 2024-06-03T06:16:01.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:01.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:16:01.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-06-03T06:16:01.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-06-03T06:16:01.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362 2-64424509453' 2024-06-03T06:16:01.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:01.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018885 2024-06-03T06:16:01.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:01.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:16:01.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018885 2024-06-03T06:16:01.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:01.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-06-03T06:16:01.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-06-03T06:16:01.464 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 128849018885 2024-06-03T06:16:01.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:16:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018884 -lt 128849018885 2024-06-03T06:16:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:16:02.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:16:02.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:16:03.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-06-03T06:16:03.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:03.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-06-03T06:16:03.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:03.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:16:03.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-06-03T06:16:03.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:03.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-06-03T06:16:03.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-06-03T06:16:03.064 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 150323855362 2024-06-03T06:16:03.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:16:03.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-06-03T06:16:03.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:03.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-06-03T06:16:03.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:03.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:16:03.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-06-03T06:16:03.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:03.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-06-03T06:16:03.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-06-03T06:16:03.355 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509453 2024-06-03T06:16:03.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:16:03.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-06-03T06:16:03.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:16:03.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:16:03.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:16:04.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:16:04.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:16:04.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:16:04.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:16:04.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:16:04.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:16:04.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:16:04.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:16:04.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:16:04.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:16:04.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:16:04.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:16:04.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:16:04.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:16:04.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:16:04.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-06-03T06:16:04.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-03T06:16:04.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-06-03T06:16:04.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-06-03T06:16:04.657 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:16:04.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-06-03T06:16:04.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-06-03T06:16:04.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:16:04.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-03T06:16:04.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-03T06:16:04.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:16:05.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-06-03T06:16:05.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-06-03T06:16:05.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:16:05.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:16:05.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:16:05.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:16:05.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-06-03T06:15:21.702705+0000 2024-06-03T06:16:05.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-06-03T06:16:05.630 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T06:16:05.631 INFO:tasks.workunit.client.0.smithi039.stdout: "deep": true, 2024-06-03T06:16:05.631 INFO:tasks.workunit.client.0.smithi039.stdout: "must": false, 2024-06-03T06:16:05.631 INFO:tasks.workunit.client.0.smithi039.stdout: "stamp": "2024-05-20T06:14:25.631519+0000" 2024-06-03T06:16:05.631 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:16:05.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-06-03T06:15:21.702705+0000 2024-06-03T06:16:05.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-03T06:16:05.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:15:21.702705+0000 2024-06-03T06:16:05.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:16:05.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:16:05.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:16:05.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:16:05.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:16:05.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:16:05.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:16:05.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:16:05.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:15:21.702705+0000 '>' 2024-06-03T06:15:21.702705+0000 2024-06-03T06:16:05.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:16:06.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:16:06.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:16:06.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:16:06.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:16:06.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:16:06.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:16:06.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:16:07.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:15:21.702705+0000 '>' 2024-06-03T06:15:21.702705+0000 2024-06-03T06:16:07.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:16:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:16:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:16:08.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:16:08.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:16:08.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:16:08.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:16:08.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:16:08.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:15:21.702705+0000 '>' 2024-06-03T06:15:21.702705+0000 2024-06-03T06:16:08.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:16:09.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:16:09.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:16:09.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:16:09.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:16:09.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:16:09.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:16:09.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:16:09.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:15:21.702705+0000 '>' 2024-06-03T06:15:21.702705+0000 2024-06-03T06:16:09.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:16:10.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:16:10.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:16:10.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:16:10.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:16:10.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:16:10.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:16:10.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:16:11.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:16:06.446742+0000 '>' 2024-06-03T06:15:21.702705+0000 2024-06-03T06:16:11.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:16:11.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-06-03T06:16:11.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:16:11.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:16:11.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:16:11.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:16:11.119 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:16:11.119 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:16:11.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:16:11.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:16:11.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:16:11.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:16:11.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:16:11.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:16:11.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:16:11.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:16:11.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:16:11.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:16:11.497 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:16:11.497 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:16:11.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:16:11.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:11.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:16:11.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018888 2024-06-03T06:16:11.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018888 2024-06-03T06:16:11.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888' 2024-06-03T06:16:11.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:11.605 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:16:11.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-06-03T06:16:11.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-06-03T06:16:11.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888 1-150323855365' 2024-06-03T06:16:11.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:11.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:16:11.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-06-03T06:16:11.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-06-03T06:16:11.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888 1-150323855365 2-64424509456' 2024-06-03T06:16:11.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:11.821 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018888 2024-06-03T06:16:11.821 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:11.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:16:11.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018888 2024-06-03T06:16:11.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:11.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018888 2024-06-03T06:16:11.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018888' 2024-06-03T06:16:11.824 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 128849018888 2024-06-03T06:16:11.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:16:12.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018888 -lt 128849018888 2024-06-03T06:16:12.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:12.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855365 2024-06-03T06:16:12.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:12.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:16:12.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855365 2024-06-03T06:16:12.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:12.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-06-03T06:16:12.115 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 150323855365 2024-06-03T06:16:12.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-06-03T06:16:12.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:16:12.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-06-03T06:16:12.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:12.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-06-03T06:16:12.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:12.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:16:12.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-06-03T06:16:12.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:12.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-06-03T06:16:12.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-06-03T06:16:12.407 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509456 2024-06-03T06:16:12.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:16:12.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-06-03T06:16:12.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:16:12.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:16:12.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:16:13.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:16:13.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:16:13.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:16:13.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:16:13.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:16:13.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:16:13.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:16:13.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:16:13.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:16:13.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:16:13.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:16:13.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:16:13.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:16:13.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:16:13.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:16:13.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-06-03T06:16:13.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:16:13.699 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:16:14.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:16:14.020 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:16:14.020 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:16:14.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:16:14.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:14.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:16:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018890 2024-06-03T06:16:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018890 2024-06-03T06:16:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890' 2024-06-03T06:16:14.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:14.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:16:14.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855366 2024-06-03T06:16:14.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855366 2024-06-03T06:16:14.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855366' 2024-06-03T06:16:14.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:14.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:16:14.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-06-03T06:16:14.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-06-03T06:16:14.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855366 2-64424509457' 2024-06-03T06:16:14.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:14.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018890 2024-06-03T06:16:14.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:14.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:16:14.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018890 2024-06-03T06:16:14.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:14.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018890 2024-06-03T06:16:14.389 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 128849018890 2024-06-03T06:16:14.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018890' 2024-06-03T06:16:14.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:16:14.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018890 2024-06-03T06:16:14.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:16:15.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:16:15.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:16:15.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018890 2024-06-03T06:16:15.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:16:16.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T06:16:16.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:16:17.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018890 -lt 128849018890 2024-06-03T06:16:17.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:17.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855366 2024-06-03T06:16:17.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:17.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:16:17.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855366 2024-06-03T06:16:17.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:17.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855366 2024-06-03T06:16:17.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855366' 2024-06-03T06:16:17.286 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 150323855366 2024-06-03T06:16:17.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:16:17.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855367 -lt 150323855366 2024-06-03T06:16:17.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:17.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-06-03T06:16:17.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:17.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:16:17.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-06-03T06:16:17.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:17.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-06-03T06:16:17.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-06-03T06:16:17.575 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509457 2024-06-03T06:16:17.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:16:17.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-06-03T06:16:17.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-03T06:16:17.870 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.187+0000 7f3ab06ed640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-03T06:16:17.870 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.187+0000 7f3ab06ed640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-03T06:16:17.871 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.187+0000 7f3ab06ed640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-06-03T06:16:17.871 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.187+0000 7f3ab06ed640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-06-03T06:16:17.871 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.187+0000 7f3ab06ed640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-03T06:16:17.871 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.445+0000 7f3ab06ed640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-06-03T06:16:17.871 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.445+0000 7f3ab06ed640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-06-03T06:16:17.871 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.445+0000 7f3ab06ed640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-03T06:16:17.871 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.445+0000 7f3ab06ed640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-06-03T06:16:17.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-06-03T06:16:17.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-06-03T06:16:18.225 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:16:18.225 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-06-03T06:16:06.446826+0000 21'10 36:96 [1,0] 1 [1,0] 1 21'10 2024-06-03T06:16:06.446742+0000 21'10 2024-06-03T06:16:06.446742+0000 0 1 periodic scrub scheduled @ 2024-06-04T06:16:06.446742+0000 10 0 2024-06-03T06:16:18.225 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:16:18.226 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:16:18.226 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:16:18.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-06-03T06:16:18.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-06-03T06:16:18.508 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:16:18.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-06-03T06:16:18.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-03T06:16:18.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-06-03T06:16:18.520 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-06-03T06:16:18.520 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:16:18.520 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-03T06:16:18.520 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-03T06:16:18.521 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:16:18.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T06:16:18.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-06-03T06:16:18.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T06:16:19.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-06-03T06:16:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:16:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:16:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:16:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:16:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-06-03T06:16:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:16:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:16:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:16:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:16:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:16:19.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:16:19.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:16:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:16:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:16:19.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:16:19.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:16:19.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-06-03T06:16:19.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:16:19.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:16:19.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:16:19.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:16:19.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:16:19.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-06-03T06:16:20.723 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:16:20.723 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:16:20.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:16:20.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:16:20.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:16:20.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:16:20.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:16:20.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:16:20.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:16:20.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:16:20.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:16:20.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:16:20.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:16:20.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:16:20.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:16:20.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:16:20.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:16:20.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:16:20.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:16:20.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:16:20.993 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:16:20.993 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:16:20.994 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:16:20.994 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:16:20.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:16:20.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:16:20.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:16:20.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:16:20.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:16:20.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:16:20.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:16:20.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:16:20.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:16:20.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:16:20.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:16:20.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:16:20.996 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:16:20.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:16:20.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:16:20.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:16:20.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:16:20.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:16:20.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:16:21.027 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:21.025+0000 7fb5502a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:16:21.029 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:21.028+0000 7fb5502a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:16:21.030 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:21.029+0000 7fb5502a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:16:21.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:16:21.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:16:21.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:16:21.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:16:21.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:16:21.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:16:21.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:16:21.296 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:16:21.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:16:21.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:16:21.547 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:21.546+0000 7fb5502a4640 -1 Falling back to public interface 2024-06-03T06:16:21.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:16:22.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:16:22.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:16:22.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:16:22.588 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:16:22.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:16:22.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:16:22.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:16:23.894 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:23.891+0000 7fb5502a4640 -1 osd.0 36 log_to_monitors true 2024-06-03T06:16:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:16:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:16:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:16:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:16:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:16:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:16:24.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:16:25.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:16:25.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:16:25.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:16:25.342 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:16:25.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:16:25.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:16:25.672 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6802/2465468139,v1:127.0.0.1:6803/2465468139] [v2:127.0.0.1:6804/2465468139,v1:127.0.0.1:6805/2465468139] exists,up f03eb923-99a6-4d3a-bfae-ada3746cb600 2024-06-03T06:16:25.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:16:25.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:16:25.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:16:25.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:16:25.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:16:25.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:16:25.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:16:25.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:16:25.673 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:16:25.673 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:16:25.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:16:25.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:16:25.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:16:25.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:16:25.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:16:25.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:16:25.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:16:25.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:16:25.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:16:26.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:16:26.060 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:16:26.060 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:16:26.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:16:26.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:26.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:16:26.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-06-03T06:16:26.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-06-03T06:16:26.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842' 2024-06-03T06:16:26.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:26.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:16:26.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855370 2024-06-03T06:16:26.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855370 2024-06-03T06:16:26.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370' 2024-06-03T06:16:26.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:26.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:16:26.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-06-03T06:16:26.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-06-03T06:16:26.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370 2-64424509461' 2024-06-03T06:16:26.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:26.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691842 2024-06-03T06:16:26.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:26.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:16:26.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691842 2024-06-03T06:16:26.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:26.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-06-03T06:16:26.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-06-03T06:16:26.441 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 171798691842 2024-06-03T06:16:26.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:16:26.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 171798691842 2024-06-03T06:16:26.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:16:27.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:16:27.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:16:28.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-06-03T06:16:28.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:28.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855370 2024-06-03T06:16:28.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:28.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:16:28.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855370 2024-06-03T06:16:28.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:28.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855370 2024-06-03T06:16:28.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855370' 2024-06-03T06:16:28.060 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 150323855370 2024-06-03T06:16:28.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:16:28.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855370 -lt 150323855370 2024-06-03T06:16:28.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:28.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-06-03T06:16:28.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:28.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:16:28.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-06-03T06:16:28.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:28.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-06-03T06:16:28.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-06-03T06:16:28.353 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509461 2024-06-03T06:16:28.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:16:28.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-06-03T06:16:28.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:16:28.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:16:28.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:16:29.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:16:29.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:16:29.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:16:29.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:16:29.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:16:29.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:16:29.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:16:29.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:16:29.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:16:29.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:16:29.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:16:29.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:16:29.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:16:29.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:16:29.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:16:29.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-06-03T06:16:29.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:16:29.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-03T06:16:29.698 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.187+0000 7f3ab06ed640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-03T06:16:29.698 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.187+0000 7f3ab06ed640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-03T06:16:29.698 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.187+0000 7f3ab06ed640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-06-03T06:16:29.699 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.187+0000 7f3ab06ed640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-06-03T06:16:29.699 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.187+0000 7f3ab06ed640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-03T06:16:29.699 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.445+0000 7f3ab06ed640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-06-03T06:16:29.699 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.445+0000 7f3ab06ed640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-06-03T06:16:29.699 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.445+0000 7f3ab06ed640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-03T06:16:29.699 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.445+0000 7f3ab06ed640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-06-03T06:16:29.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-06-03T06:16:29.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:16:29.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:16:29.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:16:29.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:16:30.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-06-03T06:16:30.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:16:30.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:16:30.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:16:30.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:16:30.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-06-03T06:16:30.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:16:30.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:16:30.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:16:30.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:16:30.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:16:30.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:16:30.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:16:30.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:16:30.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:16:30.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:16:30.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:16:30.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-06-03T06:16:30.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:16:30.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:16:30.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:16:30.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:16:30.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:16:30.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-06-03T06:16:31.677 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-06-03T06:16:31.677 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:104778fc:::obj2:head# 2024-06-03T06:16:32.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:16:32.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:16:32.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:16:32.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:16:32.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:16:32.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:16:32.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:16:32.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:16:32.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:16:32.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:16:32.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:16:32.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:16:32.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:16:32.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:16:32.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:16:32.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:16:32.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:16:32.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:16:32.202 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:16:32.202 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:16:32.202 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:16:32.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:16:32.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:16:32.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:16:32.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:16:32.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:16:32.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:16:32.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:16:32.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:16:32.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:16:32.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:16:32.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:16:32.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:16:32.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:16:32.204 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:16:32.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:16:32.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:16:32.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:16:32.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:16:32.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:16:32.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:16:32.233 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:32.233+0000 7f218b832640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:16:32.235 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:32.235+0000 7f218b832640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:16:32.237 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:32.236+0000 7f218b832640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:16:32.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:16:32.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:16:32.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:16:32.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:16:32.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:16:32.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:16:32.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:16:32.509 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:16:32.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:16:32.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:16:32.751 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:32.751+0000 7f218b832640 -1 Falling back to public interface 2024-06-03T06:16:32.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:16:33.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:16:33.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:16:33.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:16:33.798 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:16:33.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:16:33.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:16:34.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:16:35.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:16:35.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:16:35.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:16:35.105 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:16:35.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:16:35.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:16:35.150 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:35.149+0000 7f218b832640 -1 osd.1 41 log_to_monitors true 2024-06-03T06:16:35.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:16:36.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:16:36.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:16:36.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:16:36.455 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:16:36.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:16:36.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:16:36.756 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/1441953052,v1:127.0.0.1:6811/1441953052] [v2:127.0.0.1:6812/1441953052,v1:127.0.0.1:6813/1441953052] exists,up 4ce04533-c54d-430d-b5eb-85292391a87d 2024-06-03T06:16:36.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:16:36.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:16:36.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:16:36.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:16:36.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:16:36.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:16:36.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:16:36.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:16:36.757 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:16:36.757 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:16:36.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:16:36.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:16:36.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:16:36.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:16:36.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:16:36.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:16:36.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:16:36.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:16:36.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:16:37.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:16:37.140 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:16:37.140 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:16:37.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:16:37.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:37.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:16:37.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-06-03T06:16:37.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-06-03T06:16:37.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845' 2024-06-03T06:16:37.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:37.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:16:37.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528322 2024-06-03T06:16:37.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528322 2024-06-03T06:16:37.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322' 2024-06-03T06:16:37.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:37.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:16:37.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-06-03T06:16:37.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-06-03T06:16:37.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322 2-64424509464' 2024-06-03T06:16:37.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:37.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691845 2024-06-03T06:16:37.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:37.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:16:37.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691845 2024-06-03T06:16:37.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:37.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-06-03T06:16:37.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-06-03T06:16:37.467 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 171798691845 2024-06-03T06:16:37.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:16:37.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691844 -lt 171798691845 2024-06-03T06:16:37.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:16:38.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:16:38.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:16:39.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-06-03T06:16:39.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:39.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528322 2024-06-03T06:16:39.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:39.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:16:39.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528322 2024-06-03T06:16:39.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:39.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528322 2024-06-03T06:16:39.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2024-06-03T06:16:39.059 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 193273528322 2024-06-03T06:16:39.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:16:39.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528322 -lt 193273528322 2024-06-03T06:16:39.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:39.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-06-03T06:16:39.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:39.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:16:39.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-06-03T06:16:39.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:39.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-06-03T06:16:39.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-06-03T06:16:39.354 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509464 2024-06-03T06:16:39.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:16:39.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-06-03T06:16:39.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:16:39.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:16:39.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:16:40.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:16:40.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:16:40.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:16:40.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:16:40.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:16:40.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:16:40.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:16:40.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:16:40.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:16:40.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:16:40.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:16:40.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:16:40.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:16:40.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:16:40.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:16:40.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-06-03T06:16:40.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-06-03T06:16:40.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-06-03T06:16:40.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:16:40.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:16:40.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:16:40.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:16:40.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:16:06.446742+0000 2024-06-03T06:16:41.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-06-03T06:16:41.273 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 1.0 on osd.1 to repair 2024-06-03T06:16:41.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-06-03T06:16:06.446742+0000 2024-06-03T06:16:41.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-03T06:16:41.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:16:06.446742+0000 2024-06-03T06:16:41.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:16:41.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:16:41.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:16:41.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:16:41.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:16:41.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:16:41.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:16:41.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:16:41.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:16:06.446742+0000 '>' 2024-06-03T06:16:06.446742+0000 2024-06-03T06:16:41.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:16:42.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:16:42.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:16:42.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:16:42.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:16:42.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:16:42.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:16:42.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:16:42.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:16:06.446742+0000 '>' 2024-06-03T06:16:06.446742+0000 2024-06-03T06:16:42.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:16:43.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:16:43.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:16:43.875 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:16:43.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:16:43.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:16:43.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:16:43.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:16:44.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:16:06.446742+0000 '>' 2024-06-03T06:16:06.446742+0000 2024-06-03T06:16:44.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:16:45.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:16:45.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:16:45.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:16:45.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:16:45.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:16:45.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:16:45.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:16:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:16:06.446742+0000 '>' 2024-06-03T06:16:06.446742+0000 2024-06-03T06:16:45.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:16:46.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:16:46.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:16:46.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:16:46.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:16:46.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:16:46.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:16:46.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:16:46.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:16:42.166103+0000 '>' 2024-06-03T06:16:06.446742+0000 2024-06-03T06:16:46.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:16:46.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-06-03T06:16:48.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-06-03T06:16:48.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:16:48.777 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:16:49.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:16:49.103 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:16:49.103 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:16:49.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:16:49.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:49.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:16:49.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691848 2024-06-03T06:16:49.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691848 2024-06-03T06:16:49.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691848' 2024-06-03T06:16:49.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:49.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:16:49.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528325 2024-06-03T06:16:49.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528325 2024-06-03T06:16:49.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691848 1-193273528325' 2024-06-03T06:16:49.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:16:49.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:16:49.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-06-03T06:16:49.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-06-03T06:16:49.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691848 1-193273528325 2-64424509467' 2024-06-03T06:16:49.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:49.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691848 2024-06-03T06:16:49.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:49.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:16:49.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691848 2024-06-03T06:16:49.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:49.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691848 2024-06-03T06:16:49.429 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 171798691848 2024-06-03T06:16:49.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691848' 2024-06-03T06:16:49.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:16:49.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691847 -lt 171798691848 2024-06-03T06:16:49.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:16:50.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:16:50.720 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:16:51.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691849 -lt 171798691848 2024-06-03T06:16:51.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:51.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528325 2024-06-03T06:16:51.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:51.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:16:51.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528325 2024-06-03T06:16:51.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:51.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528325 2024-06-03T06:16:51.024 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 193273528325 2024-06-03T06:16:51.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528325' 2024-06-03T06:16:51.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:16:51.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528325 -lt 193273528325 2024-06-03T06:16:51.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:16:51.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-06-03T06:16:51.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:16:51.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:16:51.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-06-03T06:16:51.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:16:51.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-06-03T06:16:51.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-06-03T06:16:51.333 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509467 2024-06-03T06:16:51.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:16:51.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-06-03T06:16:51.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-06-03T06:16:51.897 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:16:51.897 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-06-03T06:16:42.166219+0000 21'10 46:133 [1,0] 1 [1,0] 1 21'10 2024-06-03T06:16:42.166103+0000 21'10 2024-06-03T06:16:42.166103+0000 0 1 periodic scrub scheduled @ 2024-06-04T06:16:42.166103+0000 9 0 2024-06-03T06:16:51.897 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:16:51.897 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:16:51.897 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:16:51.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-06-03T06:16:51.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-06-03T06:16:52.180 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:16:52.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-03T06:16:52.194 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.187+0000 7f3ab06ed640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-03T06:16:52.194 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.187+0000 7f3ab06ed640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-03T06:16:52.194 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.187+0000 7f3ab06ed640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-06-03T06:16:52.194 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.187+0000 7f3ab06ed640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-06-03T06:16:52.195 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.187+0000 7f3ab06ed640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-03T06:16:52.195 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.445+0000 7f3ab06ed640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-06-03T06:16:52.195 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.445+0000 7f3ab06ed640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-06-03T06:16:52.195 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.445+0000 7f3ab06ed640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-03T06:16:52.195 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:06.445+0000 7f3ab06ed640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-06-03T06:16:52.195 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:42.165+0000 7f216ed95640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR planned AUTO_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-06-03T06:16:52.195 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:42.165+0000 7f216ed95640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-03T06:16:52.195 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:42.165+0000 7f216ed95640 15 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-06-03T06:16:52.195 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:42.165+0000 7f216ed95640 20 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-06-03T06:16:52.195 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:16:42.165+0000 7f216ed95640 19 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-03T06:16:52.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T06:16:52.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:16:52.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:16:52.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:16:52.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:16:52.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:16:52.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:16:52.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:16:52.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:16:52.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:16:52.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:16:52.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:16:52.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:16:52.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:16:52.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:16:52.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:16:52.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:16:52.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:16:52.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:16:52.307 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:16:52.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:16:52.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:16:52.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:16:52.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:16:52.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:16:52.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:16:52.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:16:52.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:16:52.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:16:52.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T06:16:52.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T06:16:52.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T06:16:52.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T06:16:52.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:16:52.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:16:52.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:16:52.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:16:52.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:16:52.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:16:52.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:16:52.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:16:52.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:16:52.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:16:52.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:16:52.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:16:52.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:16:52.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:16:52.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:16:52.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:16:52.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:16:52.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:16:52.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:16:52.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:16:52.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:16:52.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:16:52.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:16:52.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:16:52.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:16:52.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:16:52.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:16:52.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:16:52.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T06:16:52.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:16:52.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:16:52.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:16:52.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T06:16:52.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:16:52.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T06:16:52.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:16:52.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T06:16:52.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-06-03T06:16:52.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-06-03T06:16:52.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-06-03T06:16:52.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-06-03T06:16:52.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T06:16:52.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T06:16:52.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T06:16:52.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T06:16:52.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T06:16:52.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T06:16:52.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T06:16:52.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:16:52.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:16:52.746 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:16:52.746 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:16:52.746 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:16:52.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:16:52.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T06:16:52.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T06:16:52.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T06:16:52.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:16:52.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:16:52.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T06:16:52.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T06:16:52.948 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:16:52.948 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:16:52.948 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:16:52.948 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:16:52.948 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:16:52.948 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:16:52.948 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:16:52.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:16:52.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T06:16:53.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T06:16:53.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:16:53.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:16:53.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T06:16:53.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T06:16:53.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:16:53.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:16:53.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:16:53.032 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:16:53.032 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:16:53.032 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:16:53.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:16:53.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:16:53.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T06:16:53.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-06-03T06:16:53.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T06:16:53.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T06:16:53.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T06:16:53.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T06:16:53.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T06:16:53.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T06:16:53.932 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T06:16:53.932 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:16:53.932 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:16:53.932 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:16:53.932 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:16:53.932 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:16:53.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:16:53.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:16:53.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:16:53.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:16:53.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-06-03T06:16:53.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-06-03T06:16:53.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:16:53.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:16:53.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:16:53.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T06:16:53.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:16:53.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:16:53.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:16:53.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:16:53.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:16:53.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:16:53.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:16:53.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:16:53.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:16:53.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:16:53.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:16:53.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:16:53.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:16:53.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:16:53.970 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:16:53.971 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:16:53.971 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:16:53.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:16:53.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:16:53.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:16:53.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:16:53.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:16:53.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:16:53.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:16:53.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:16:53.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:16:53.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:16:53.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:16:53.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:16:53.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:16:53.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:16:53.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:16:53.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=65ae30e5-fe18-4882-a9cd-72e979b31caa 2024-06-03T06:16:53.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 65ae30e5-fe18-4882-a9cd-72e979b31caa' 2024-06-03T06:16:53.974 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 65ae30e5-fe18-4882-a9cd-72e979b31caa 2024-06-03T06:16:53.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:16:53.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDVX11mcb2mOhAAFB7CxFwe2OFzIzcsFEJCrQ== 2024-06-03T06:16:53.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDVX11mcb2mOhAAFB7CxFwe2OFzIzcsFEJCrQ=="}' 2024-06-03T06:16:53.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 65ae30e5-fe18-4882-a9cd-72e979b31caa -i td/osd-scrub-repair/0/new.json 2024-06-03T06:16:54.195 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:16:54.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T06:16:54.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDVX11mcb2mOhAAFB7CxFwe2OFzIzcsFEJCrQ== --osd-uuid 65ae30e5-fe18-4882-a9cd-72e979b31caa 2024-06-03T06:16:54.238 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:54.238+0000 7fb622432640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:16:54.247 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:54.246+0000 7fb622432640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:16:54.248 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:54.248+0000 7fb622432640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:16:54.249 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:54.248+0000 7fb622432640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:16:54.249 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:54.248+0000 7fb622432640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:16:54.249 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:54.248+0000 7fb622432640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:16:54.339 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:16:54.338+0000 7fb622432640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T06:16:59.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T06:16:59.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:16:59.864 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:16:59.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T06:16:59.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:17:00.251 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:17:00.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T06:17:00.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:17:00.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:17:00.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:17:00.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:17:00.281 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:00.280+0000 7f744ae61640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:17:00.290 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:00.290+0000 7f744ae61640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:17:00.292 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:00.292+0000 7f744ae61640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:17:00.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T06:17:00.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:17:00.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:17:00.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:17:00.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:17:00.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:00.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:17:00.540 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:17:00.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:00.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:00.804 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:00.804+0000 7f744ae61640 -1 Falling back to public interface 2024-06-03T06:17:00.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:01.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:01.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:01.830 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:17:01.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:17:01.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:01.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:02.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:03.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:03.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:03.134 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:17:03.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:17:03.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:03.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:03.166 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:03.165+0000 7f744ae61640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:17:03.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:04.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:04.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:04.593 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:17:04.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:17:04.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:04.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:04.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:05.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:05.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:05.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:17:05.941 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:17:05.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:05.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:06.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:07.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:07.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:07.244 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:17:07.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:17:07.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:07.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:07.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:08.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:08.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:08.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:17:08.544 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:17:08.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:08.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:08.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:09.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:09.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:09.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:17:09.873 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:17:09.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:09.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:10.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:11.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:11.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:11.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:17:11.178 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:17:11.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:11.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:11.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:12.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:12.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:12.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:17:12.477 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:17:12.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:12.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:12.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:13.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:13.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:13.859 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:17:13.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:17:13.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:13.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:14.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:15.158 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:17:15.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:15.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:17:15.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:15.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:15.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:16.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:16.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:16.766 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:17:16.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:17:16.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:16.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:17.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:18.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:18.166 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:17:18.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:17:18.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:18.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:18.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:19.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:19.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:19.830 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:17:19.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:17:19.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:19.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:20.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:21.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:21.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:21.265 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:17:21.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:17:21.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:21.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:21.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:22.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:22.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:22.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:17:22.951 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:17:22.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:22.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:23.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:24.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:24.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:24.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:17:24.506 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:17:24.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:24.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:24.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:25.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:25.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:25.924 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:17:25.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:17:25.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:25.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:17:26.228 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1196863623,v1:127.0.0.1:6803/1196863623] [v2:127.0.0.1:6804/1196863623,v1:127.0.0.1:6805/1196863623] exists,up 65ae30e5-fe18-4882-a9cd-72e979b31caa 2024-06-03T06:17:26.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:17:26.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:17:26.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:17:26.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-06-03T06:17:26.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:17:26.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:17:26.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:17:26.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T06:17:26.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:17:26.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:17:26.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:17:26.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:17:26.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:17:26.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:17:26.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:17:26.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:17:26.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:17:26.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:17:26.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:17:26.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:17:26.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:17:26.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:17:26.239 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:17:26.239 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:17:26.239 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:17:26.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:17:26.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:17:26.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:17:26.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:17:26.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:17:26.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:17:26.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:17:26.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:17:26.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:17:26.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:17:26.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:17:26.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:17:26.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:17:26.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:17:26.240 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:17:26.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a5895f8c-0ac7-4475-9b95-a44e183c1dee 2024-06-03T06:17:26.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a5895f8c-0ac7-4475-9b95-a44e183c1dee' 2024-06-03T06:17:26.241 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 a5895f8c-0ac7-4475-9b95-a44e183c1dee 2024-06-03T06:17:26.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:17:26.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD2X11m/Zv7DhAAB0w/HMEN8juseI/QHRgqPA== 2024-06-03T06:17:26.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD2X11m/Zv7DhAAB0w/HMEN8juseI/QHRgqPA=="}' 2024-06-03T06:17:26.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a5895f8c-0ac7-4475-9b95-a44e183c1dee -i td/osd-scrub-repair/1/new.json 2024-06-03T06:17:26.641 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:17:26.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T06:17:26.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD2X11m/Zv7DhAAB0w/HMEN8juseI/QHRgqPA== --osd-uuid a5895f8c-0ac7-4475-9b95-a44e183c1dee 2024-06-03T06:17:26.685 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:26.684+0000 7f208b00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:17:26.717 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:26.717+0000 7f208b00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:17:26.718 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:26.718+0000 7f208b00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:17:26.719 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:26.719+0000 7f208b00f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:17:26.719 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:26.719+0000 7f208b00f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:17:26.719 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:26.719+0000 7f208b00f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:17:26.784 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:26.784+0000 7f208b00f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T06:17:32.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T06:17:32.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:17:32.852 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:17:32.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T06:17:32.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:17:33.227 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:17:33.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T06:17:33.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:17:33.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:17:33.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:17:33.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:17:33.257 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:33.256+0000 7f29998a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:17:33.259 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:33.259+0000 7f29998a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:17:33.260 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:33.260+0000 7f29998a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:17:33.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T06:17:33.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:17:33.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:17:33.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:17:33.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:17:33.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:33.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:17:33.521 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:17:33.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:33.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:33.778 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:33.778+0000 7f29998a8640 -1 Falling back to public interface 2024-06-03T06:17:33.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:34.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:34.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:34.817 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:17:34.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:17:34.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:34.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:35.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:36.111 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:17:36.111+0000 7f29998a8640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:17:36.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:36.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:36.126 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:17:36.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:17:36.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:36.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:36.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:37.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:37.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:37.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:17:37.480 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:17:37.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:37.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:37.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:38.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:38.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:38.799 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:17:38.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:17:38.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:38.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:39.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:40.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:40.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:40.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:17:40.178 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:17:40.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:40.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:40.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:41.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:41.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:41.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:17:41.483 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:17:41.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:41.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:41.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:42.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:42.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:42.816 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:17:42.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:17:42.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:42.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:43.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:44.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:44.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:44.301 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:17:44.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:17:44.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:44.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:44.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:45.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:45.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:45.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:17:45.601 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:17:45.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:45.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:46.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:47.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:47.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:47.006 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:17:47.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:17:47.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:47.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:47.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:48.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:48.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:48.396 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:17:48.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:17:48.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:48.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:48.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:49.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:49.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:49.696 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:17:49.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:17:49.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:49.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:50.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:51.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:51.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:51.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:17:51.250 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:17:51.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:51.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:51.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:52.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:52.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:52.874 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:17:52.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:17:52.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:52.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:53.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:54.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:54.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:54.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:17:54.250 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:17:54.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:54.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:54.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:55.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:55.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:55.960 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:17:55.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:17:55.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:55.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:56.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:57.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:57.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:57.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:17:57.724 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:17:57.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:57.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:58.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:17:59.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:17:59.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:17:59.333 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:17:59.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:17:59.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:17:59.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:17:59.643 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2254604496,v1:127.0.0.1:6811/2254604496] [v2:127.0.0.1:6812/2254604496,v1:127.0.0.1:6813/2254604496] exists,up a5895f8c-0ac7-4475-9b95-a44e183c1dee 2024-06-03T06:17:59.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:17:59.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:17:59.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:17:59.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-06-03T06:17:59.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:17:59.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:17:59.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:17:59.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T06:17:59.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:17:59.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:17:59.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:17:59.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:17:59.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:17:59.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:17:59.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:17:59.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:17:59.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:17:59.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:17:59.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:17:59.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:17:59.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:17:59.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:17:59.650 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:17:59.651 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:17:59.654 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:17:59.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:17:59.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:17:59.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:17:59.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:17:59.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:17:59.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:17:59.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:17:59.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:17:59.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:17:59.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:17:59.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:17:59.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:17:59.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:17:59.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:17:59.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:17:59.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=75a11e1a-d3ac-46c2-98da-6548c47897d9 2024-06-03T06:17:59.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 75a11e1a-d3ac-46c2-98da-6548c47897d9' 2024-06-03T06:17:59.656 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 75a11e1a-d3ac-46c2-98da-6548c47897d9 2024-06-03T06:17:59.657 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:17:59.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAXYF1mc6qrJxAAztT4YNa+xJyLK1XelbTp8A== 2024-06-03T06:17:59.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAXYF1mc6qrJxAAztT4YNa+xJyLK1XelbTp8A=="}' 2024-06-03T06:17:59.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 75a11e1a-d3ac-46c2-98da-6548c47897d9 -i td/osd-scrub-repair/2/new.json 2024-06-03T06:18:00.001 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:18:00.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-03T06:18:00.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAXYF1mc6qrJxAAztT4YNa+xJyLK1XelbTp8A== --osd-uuid 75a11e1a-d3ac-46c2-98da-6548c47897d9 2024-06-03T06:18:00.045 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:00.044+0000 7f0579e1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:18:00.086 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:00.086+0000 7f0579e1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:18:00.087 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:00.087+0000 7f0579e1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:18:00.087 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:00.087+0000 7f0579e1e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:18:00.087 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:00.087+0000 7f0579e1e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:18:00.087 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:00.087+0000 7f0579e1e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:18:00.169 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:00.169+0000 7f0579e1e640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T06:18:06.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-03T06:18:06.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:18:06.202 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T06:18:06.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T06:18:06.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:18:06.581 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:18:06.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T06:18:06.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:18:06.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:18:06.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:18:06.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:18:06.610 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:06.609+0000 7f90a2701640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:18:06.612 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:06.611+0000 7f90a2701640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:18:06.613 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:06.612+0000 7f90a2701640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:18:06.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T06:18:06.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:18:06.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:18:06.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:18:06.871 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:18:06.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:18:06.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:06.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:18:06.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:06.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:07.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:07.139+0000 7f90a2701640 -1 Falling back to public interface 2024-06-03T06:18:07.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:08.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:08.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:08.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:18:08.199 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:18:08.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:08.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:08.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:09.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:09.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:09.498 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:18:09.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:18:09.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:09.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:09.577 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:09.575+0000 7f90a2701640 -1 osd.2 0 log_to_monitors true 2024-06-03T06:18:09.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:10.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:10.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:10.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:18:10.804 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:18:10.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:10.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:11.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:12.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:12.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:12.108 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:18:12.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:18:12.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:12.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:12.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:13.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:13.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:13.409 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:18:13.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:18:13.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:13.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:13.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:14.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:14.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:14.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:18:14.812 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:18:14.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:14.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:15.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:16.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:16.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:16.112 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:18:16.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:18:16.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:16.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:16.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:17.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:17.416 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:18:17.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:17.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:18:17.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:17.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:17.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:18.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:18.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:18.798 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:18:18.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:18:18.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:18.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:19.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:20.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:20.108 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:18:20.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:20.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:18:20.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:20.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:20.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:21.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:21.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:21.412 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:18:21.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:18:21.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:21.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:21.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:22.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:22.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:22.747 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:18:22.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:18:22.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:22.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:23.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:24.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:24.104 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:18:24.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:24.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:18:24.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:24.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:24.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:25.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:25.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:25.829 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:18:25.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:18:25.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:25.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:26.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:27.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:27.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:27.285 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:18:27.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:18:27.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:27.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:27.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:28.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:28.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:28.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:18:28.856 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:18:28.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:28.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:29.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:30.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:30.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:30.366 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:18:30.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:18:30.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:30.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:30.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:31.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:31.893 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:18:31.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:31.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:18:31.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:31.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:32.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:33.361 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:18:33.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:18:33.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:33.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:18:33.657 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2907106304,v1:127.0.0.1:6819/2907106304] [v2:127.0.0.1:6820/2907106304,v1:127.0.0.1:6821/2907106304] exists,up 75a11e1a-d3ac-46c2-98da-6548c47897d9 2024-06-03T06:18:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:18:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:18:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:18:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-06-03T06:18:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-06-03T06:18:34.206 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'testpool' created 2024-06-03T06:18:34.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:18:35.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-06-03T06:18:35.848 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 2 2024-06-03T06:18:35.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-06-03T06:18:35.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:18:35.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:18:35.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:18:35.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:18:35.861 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:18:35.862 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:18:35.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:18:35.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:18:35.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:18:35.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:18:35.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:18:35.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:18:35.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:18:35.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:18:35.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:18:36.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:18:36.397 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:18:36.397 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:18:36.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:18:36.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:18:36.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:18:36.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-06-03T06:18:36.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-06-03T06:18:36.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-06-03T06:18:36.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:18:36.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:18:36.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-06-03T06:18:36.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-06-03T06:18:36.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672972' 2024-06-03T06:18:36.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:18:36.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:18:36.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542148 2024-06-03T06:18:36.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542148 2024-06-03T06:18:36.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672972 2-60129542148' 2024-06-03T06:18:36.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:18:36.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-06-03T06:18:36.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:18:36.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:18:36.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-06-03T06:18:36.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:18:36.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-06-03T06:18:36.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-06-03T06:18:36.769 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836498 2024-06-03T06:18:36.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:18:37.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-06-03T06:18:37.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:18:38.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:18:38.068 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:18:38.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-06-03T06:18:38.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:18:38.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-06-03T06:18:38.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:18:38.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:18:38.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-06-03T06:18:38.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:18:38.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-06-03T06:18:38.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-06-03T06:18:38.372 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672972 2024-06-03T06:18:38.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:18:38.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-06-03T06:18:38.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:18:38.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542148 2024-06-03T06:18:38.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:18:38.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:18:38.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542148 2024-06-03T06:18:38.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:18:38.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542148 2024-06-03T06:18:38.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2024-06-03T06:18:38.662 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 60129542148 2024-06-03T06:18:38.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:18:38.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542148 -lt 60129542148 2024-06-03T06:18:38.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:18:38.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:18:38.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:18:39.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:18:39.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:18:39.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:18:39.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:18:39.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:18:39.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:18:39.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:18:39.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:18:39.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T06:18:39.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:18:39.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:18:39.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:18:39.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-03T06:18:39.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-03T06:18:39.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:18:39.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-03T06:18:39.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:18:39.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:18:40.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:18:40.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:18:40.068 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:18:40.068 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:18:40.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:18:40.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:18:40.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:18:40.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:18:40.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:18:40.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:18:40.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:18:40.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:18:40.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:18:40.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:18:40.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:18:40.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-06-03T06:18:40.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-06-03T06:18:40.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-06-03T06:18:40.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-03T06:18:40.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:18:40.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-03T06:18:40.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:18:41.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-03T06:18:41.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:18:41.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-03T06:18:41.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:18:41.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-03T06:18:41.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:18:41.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-03T06:18:41.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:18:41.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-03T06:18:41.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:18:41.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-03T06:18:41.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:18:41.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-03T06:18:41.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:18:42.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-03T06:18:42.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:18:42.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-06-03T06:18:42.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-03T06:18:42.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:18:42.146 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-03T06:18:42.146 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:18:42.146 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:18:42.146 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:18:42.146 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:18:42.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T06:18:42.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:18:42.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T06:18:42.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-06-03T06:18:42.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:18:42.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:18:42.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:18:42.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:18:42.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-06-03T06:18:42.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:18:42.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:18:42.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:18:42.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:18:42.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:18:42.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:18:42.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:18:42.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:18:42.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:18:42.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:18:43.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:18:43.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-06-03T06:18:43.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:18:43.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:18:43.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:18:43.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:18:43.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:18:43.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-06-03T06:18:44.340 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:ff7b1f36:::obj1:head# 2024-06-03T06:18:44.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:18:44.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:18:44.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:18:44.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:18:44.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:18:44.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:18:44.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:18:44.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:18:44.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:18:44.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:18:44.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:18:44.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:18:44.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:18:44.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:18:44.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:18:44.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:18:44.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:18:44.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:18:44.860 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:18:44.860 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:18:44.860 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:18:44.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:18:44.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:18:44.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:18:44.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:18:44.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:18:44.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:18:44.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:18:44.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:18:44.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:18:44.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:18:44.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:18:44.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:18:44.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:18:44.862 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:18:44.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:18:44.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:18:44.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:18:44.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:18:44.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:18:44.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:18:44.893 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:44.892+0000 7fa46342e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:18:44.895 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:44.895+0000 7fa46342e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:18:44.897 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:44.896+0000 7fa46342e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:18:45.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:18:45.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:18:45.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:18:45.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:18:45.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:18:45.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:45.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:18:45.166 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:18:45.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:45.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:18:45.418 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:45.418+0000 7fa46342e640 -1 Falling back to public interface 2024-06-03T06:18:45.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:46.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:46.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:46.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:18:46.495 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:18:46.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:46.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:18:46.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:47.777 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:47.777+0000 7fa46342e640 -1 osd.0 20 log_to_monitors true 2024-06-03T06:18:47.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:47.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:47.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:18:47.796 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:18:47.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:47.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:18:48.241 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/1809578294,v1:127.0.0.1:6803/1809578294] [v2:127.0.0.1:6804/1809578294,v1:127.0.0.1:6805/1809578294] exists,up 65ae30e5-fe18-4882-a9cd-72e979b31caa 2024-06-03T06:18:48.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:18:48.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:18:48.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:18:48.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:18:48.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:18:48.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:18:48.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:18:48.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:18:48.242 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:18:48.242 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:18:48.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:18:48.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:18:48.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:18:48.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:18:48.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:18:48.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:18:48.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:18:48.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:18:48.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:18:48.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:18:48.627 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:18:48.627 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:18:48.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:18:48.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:18:48.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:18:48.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215106 2024-06-03T06:18:48.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215106 2024-06-03T06:18:48.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106' 2024-06-03T06:18:48.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:18:48.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:18:48.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-06-03T06:18:48.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-06-03T06:18:48.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672975' 2024-06-03T06:18:48.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:18:48.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:18:49.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542152 2024-06-03T06:18:49.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542152 2024-06-03T06:18:49.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672975 2-60129542152' 2024-06-03T06:18:49.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:18:49.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215106 2024-06-03T06:18:49.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:18:49.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:18:49.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215106 2024-06-03T06:18:49.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:18:49.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215106 2024-06-03T06:18:49.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-06-03T06:18:49.089 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 103079215106 2024-06-03T06:18:49.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:18:49.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 103079215106 2024-06-03T06:18:49.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:18:50.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:18:50.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:18:50.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215106 -lt 103079215106 2024-06-03T06:18:50.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:18:50.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-06-03T06:18:50.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:18:50.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:18:50.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-06-03T06:18:50.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:18:50.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-06-03T06:18:50.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-06-03T06:18:50.682 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672975 2024-06-03T06:18:50.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:18:50.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672975 2024-06-03T06:18:50.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:18:50.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542152 2024-06-03T06:18:50.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:18:50.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:18:50.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542152 2024-06-03T06:18:50.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:18:50.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542152 2024-06-03T06:18:50.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542152' 2024-06-03T06:18:50.978 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 60129542152 2024-06-03T06:18:50.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:18:51.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542152 -lt 60129542152 2024-06-03T06:18:51.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:18:51.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:18:51.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:18:51.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:18:51.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:18:51.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:18:51.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:18:51.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:18:51.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:18:51.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:18:51.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:18:51.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:18:51.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:18:51.929 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:18:51.929 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:18:52.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:18:52.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:18:52.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:18:52.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-06-03T06:18:52.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:18:52.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:18:52.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:18:52.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:18:52.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-06-03T06:18:52.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:18:52.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:18:52.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:18:52.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:18:52.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-06-03T06:18:52.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:18:52.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:18:52.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:18:52.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:18:52.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:18:52.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:18:52.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:18:52.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:18:52.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:18:52.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:18:52.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:18:52.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-06-03T06:18:52.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:18:52.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:18:52.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:18:52.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:18:52.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:18:52.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-06-03T06:18:54.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:18:54.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:18:54.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:18:54.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:18:54.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:18:54.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:18:54.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:18:54.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:18:54.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:18:54.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:18:54.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:18:54.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:18:54.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:18:54.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:18:54.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:18:54.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:18:54.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:18:54.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:18:54.788 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:18:54.788 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:18:54.788 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:18:54.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:18:54.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:18:54.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:18:54.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:18:54.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:18:54.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:18:54.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:18:54.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:18:54.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:18:54.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:18:54.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:18:54.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:18:54.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:18:54.790 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:18:54.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:18:54.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:18:54.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:18:54.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:18:54.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:18:54.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:18:54.821 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:54.820+0000 7feeaf215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:18:54.823 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:54.822+0000 7feeaf215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:18:54.825 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:54.824+0000 7feeaf215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:18:55.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:18:55.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:18:55.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:18:55.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:18:55.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:18:55.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:55.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:18:55.130 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:18:55.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:55.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:18:55.346 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:55.345+0000 7feeaf215640 -1 Falling back to public interface 2024-06-03T06:18:55.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:56.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:56.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:56.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:18:56.427 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:18:56.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:56.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:18:56.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:18:57.680 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:18:57.679+0000 7feeaf215640 -1 osd.1 25 log_to_monitors true 2024-06-03T06:18:57.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:18:57.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:18:57.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:18:57.729 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:18:57.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:18:57.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:18:58.104 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 29 up_thru 24 down_at 26 last_clean_interval [10,25) [v2:127.0.0.1:6810/703293896,v1:127.0.0.1:6811/703293896] [v2:127.0.0.1:6812/703293896,v1:127.0.0.1:6813/703293896] exists,up a5895f8c-0ac7-4475-9b95-a44e183c1dee 2024-06-03T06:18:58.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:18:58.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:18:58.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:18:58.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:18:58.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:18:58.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:18:58.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:18:58.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:18:58.105 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:18:58.105 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:18:58.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:18:58.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:18:58.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:18:58.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:18:58.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:18:58.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:18:58.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:18:58.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:18:58.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:18:58.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:18:58.504 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:18:58.504 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:18:58.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:18:58.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:18:58.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:18:58.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215109 2024-06-03T06:18:58.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215109 2024-06-03T06:18:58.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109' 2024-06-03T06:18:58.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:18:58.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:18:58.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-06-03T06:18:58.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-06-03T06:18:58.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109 1-124554051586' 2024-06-03T06:18:58.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:18:58.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:18:58.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542155 2024-06-03T06:18:58.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542155 2024-06-03T06:18:58.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109 1-124554051586 2-60129542155' 2024-06-03T06:18:58.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:18:58.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215109 2024-06-03T06:18:58.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:18:58.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:18:58.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215109 2024-06-03T06:18:58.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:18:58.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215109 2024-06-03T06:18:58.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215109' 2024-06-03T06:18:58.849 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 103079215109 2024-06-03T06:18:58.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:18:59.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215108 -lt 103079215109 2024-06-03T06:18:59.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:19:00.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:19:00.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:19:00.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215109 -lt 103079215109 2024-06-03T06:19:00.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:19:00.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-124554051586 2024-06-03T06:19:00.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:19:00.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:19:00.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-124554051586 2024-06-03T06:19:00.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:19:00.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-06-03T06:19:00.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-06-03T06:19:00.443 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 124554051586 2024-06-03T06:19:00.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:19:00.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-06-03T06:19:00.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:19:00.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542155 2024-06-03T06:19:00.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:19:00.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:19:00.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542155 2024-06-03T06:19:00.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:19:00.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542155 2024-06-03T06:19:00.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2024-06-03T06:19:00.779 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 60129542155 2024-06-03T06:19:00.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:19:01.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542155 -lt 60129542155 2024-06-03T06:19:01.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:19:01.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:19:01.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:19:01.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:19:01.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:19:01.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:19:01.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:19:01.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:19:01.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:19:01.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:19:01.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:19:01.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T06:19:01.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:19:01.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:19:01.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:19:02.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-03T06:19:02.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-03T06:19:02.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:19:02.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-03T06:19:02.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:19:02.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:19:02.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:19:02.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:19:02.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:19:02.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:19:02.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:19:02.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:19:02.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:19:02.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:19:02.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T06:19:02.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:19:02.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:19:02.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:19:02.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-03T06:19:02.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-03T06:19:02.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:19:02.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:19:02.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:19:02.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:19:02.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:19:02.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:19:02.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:19:03.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T06:19:03.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T06:19:03.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T06:19:03.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:19:03.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T06:19:03.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:19:03.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:19:03.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:19:03.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:19:03.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:19:03.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:19:03.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:19:03.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:19:03.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T06:19:03.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:19:03.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:19:03.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:19:04.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-03T06:19:04.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-03T06:19:04.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:19:04.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:19:04.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:19:04.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:19:04.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:19:04.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:19:04.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:19:04.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T06:19:04.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T06:19:04.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-03T06:19:04.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:19:04.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-03T06:19:04.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:19:04.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:19:04.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:19:04.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:19:04.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:19:04.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:19:04.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:19:04.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:19:05.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:19:05.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:19:05.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:19:05.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:19:05.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:19:05.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:19:05.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:19:05.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-06-03T06:19:05.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-03T06:19:05.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:19:05.505 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-03T06:19:05.505 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:19:05.505 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:19:05.505 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:19:05.505 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:19:05.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T06:19:05.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:19:05.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T06:19:06.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-06-03T06:19:06.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:19:06.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:19:06.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:19:06.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:19:06.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-06-03T06:19:06.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:19:06.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:19:06.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:19:06.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:19:06.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:19:06.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:19:06.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:19:06.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:19:06.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:19:06.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:19:06.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:19:06.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-06-03T06:19:06.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:19:06.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:19:06.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:19:06.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:19:06.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:19:06.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-06-03T06:19:07.600 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:104778fc:::obj2:head# 2024-06-03T06:19:08.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:19:08.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:19:08.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:19:08.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:19:08.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:19:08.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:19:08.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:19:08.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:19:08.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:19:08.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:19:08.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:19:08.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:19:08.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:19:08.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:19:08.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:19:08.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:19:08.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:19:08.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:19:08.123 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:19:08.123 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:19:08.123 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:19:08.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:19:08.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:19:08.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:19:08.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:19:08.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:19:08.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:19:08.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:19:08.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:19:08.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:19:08.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:19:08.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:19:08.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:19:08.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:19:08.125 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:19:08.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:19:08.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:19:08.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:19:08.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:19:08.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:19:08.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:19:08.155 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:08.155+0000 7f15df00e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:19:08.158 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:08.158+0000 7f15df00e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:19:08.160 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:08.159+0000 7f15df00e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:19:08.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:19:08.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:19:08.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:19:08.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:19:08.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:19:08.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:19:08.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:19:08.451 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:19:08.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:19:08.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:19:08.672 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:08.672+0000 7f15df00e640 -1 Falling back to public interface 2024-06-03T06:19:08.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:19:09.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:19:09.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:19:09.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:19:09.740 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:19:09.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:19:09.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:19:10.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:19:11.008 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:11.008+0000 7f15df00e640 -1 osd.0 30 log_to_monitors true 2024-06-03T06:19:11.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:19:11.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:19:11.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:19:11.043 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:19:11.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:19:11.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:19:11.442 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 34 up_thru 26 down_at 31 last_clean_interval [24,30) [v2:127.0.0.1:6802/4280741197,v1:127.0.0.1:6803/4280741197] [v2:127.0.0.1:6804/4280741197,v1:127.0.0.1:6805/4280741197] exists,up 65ae30e5-fe18-4882-a9cd-72e979b31caa 2024-06-03T06:19:11.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:19:11.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:19:11.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:19:11.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:19:11.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:19:11.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:19:11.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:19:11.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:19:11.443 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:19:11.443 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:19:11.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:19:11.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:19:11.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:19:11.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:19:11.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:19:11.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:19:11.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:19:11.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:19:11.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:19:11.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:19:11.926 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:19:11.927 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:19:11.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:19:11.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:19:11.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:19:12.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-06-03T06:19:12.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-06-03T06:19:12.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066' 2024-06-03T06:19:12.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:19:12.035 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:19:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-06-03T06:19:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-06-03T06:19:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-124554051589' 2024-06-03T06:19:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:19:12.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:19:12.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542158 2024-06-03T06:19:12.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542158 2024-06-03T06:19:12.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-124554051589 2-60129542158' 2024-06-03T06:19:12.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:19:12.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888066 2024-06-03T06:19:12.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:19:12.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:19:12.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888066 2024-06-03T06:19:12.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:19:12.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-06-03T06:19:12.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-06-03T06:19:12.255 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 146028888066 2024-06-03T06:19:12.255 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:19:12.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888066 -lt 146028888066 2024-06-03T06:19:12.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:19:12.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-124554051589 2024-06-03T06:19:12.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:19:12.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:19:12.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-124554051589 2024-06-03T06:19:12.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:19:12.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-06-03T06:19:12.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 124554051589' 2024-06-03T06:19:12.546 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 124554051589 2024-06-03T06:19:12.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:19:12.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051589 2024-06-03T06:19:12.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:19:13.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:19:13.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:19:14.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051589 2024-06-03T06:19:14.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:19:15.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T06:19:15.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:19:15.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051590 -lt 124554051589 2024-06-03T06:19:15.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:19:15.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542158 2024-06-03T06:19:15.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:19:15.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:19:15.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542158 2024-06-03T06:19:15.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:19:15.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542158 2024-06-03T06:19:15.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542158' 2024-06-03T06:19:15.451 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 60129542158 2024-06-03T06:19:15.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:19:15.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542159 -lt 60129542158 2024-06-03T06:19:15.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:19:15.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:19:15.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:19:16.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:19:16.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:19:16.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:19:16.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:19:16.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:19:16.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:19:16.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:19:16.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:19:16.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:19:16.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:19:16.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:19:16.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:19:16.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:19:16.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:19:16.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:19:16.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-06-03T06:19:16.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:19:16.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:19:16.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:19:16.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:19:17.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-03T06:19:17.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:19:17.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:19:17.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:19:17.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:19:17.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-03T06:19:17.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:19:17.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:19:17.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:19:17.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:19:17.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:19:17.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:19:17.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:19:17.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:19:17.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:19:17.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:19:17.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:19:17.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-03T06:19:17.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:19:17.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:19:17.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:19:17.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:19:17.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:19:17.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-06-03T06:19:18.688 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-06-03T06:19:19.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:19:19.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:19:19.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:19:19.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:19:19.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:19:19.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:19:19.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:19:19.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:19:19.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:19:19.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:19:19.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:19:19.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:19:19.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:19:19.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:19:19.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:19:19.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:19:19.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:19:19.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:19:19.208 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:19:19.209 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:19:19.209 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:19:19.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:19:19.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:19:19.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:19:19.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:19:19.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:19:19.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:19:19.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:19:19.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:19:19.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:19:19.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:19:19.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:19:19.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:19:19.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:19:19.211 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:19:19.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:19:19.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:19:19.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:19:19.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:19:19.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:19:19.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:19:19.241 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:19.241+0000 7fb42f901640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:19:19.244 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:19.243+0000 7fb42f901640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:19:19.245 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:19.245+0000 7fb42f901640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:19:19.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:19:19.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:19:19.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:19:19.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:19:19.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:19:19.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:19:19.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:19:19.512 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:19:19.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:19:19.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:19:19.762 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:19.762+0000 7fb42f901640 -1 Falling back to public interface 2024-06-03T06:19:19.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:19:20.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:19:20.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:19:20.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:19:20.807 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:19:20.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:19:20.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:19:21.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:19:22.094 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:22.094+0000 7fb42f901640 -1 osd.1 35 log_to_monitors true 2024-06-03T06:19:22.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:19:22.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:19:22.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:19:22.139 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:19:22.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:19:22.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:19:22.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:19:23.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:19:23.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:19:23.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:19:23.476 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:19:23.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:19:23.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:19:23.775 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [29,35) [v2:127.0.0.1:6810/3730303010,v1:127.0.0.1:6811/3730303010] [v2:127.0.0.1:6812/3730303010,v1:127.0.0.1:6813/3730303010] exists,up a5895f8c-0ac7-4475-9b95-a44e183c1dee 2024-06-03T06:19:23.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:19:23.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:19:23.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:19:23.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:19:23.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:19:23.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:19:23.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:19:23.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:19:23.776 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:19:23.776 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:19:23.777 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:19:23.777 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:19:23.777 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:19:23.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:19:23.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:19:23.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:19:23.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:19:23.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:19:23.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:19:24.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:19:24.171 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:19:24.171 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:19:24.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:19:24.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:19:24.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:19:24.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888069 2024-06-03T06:19:24.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888069 2024-06-03T06:19:24.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888069' 2024-06-03T06:19:24.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:19:24.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:19:24.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-06-03T06:19:24.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-06-03T06:19:24.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888069 1-167503724546' 2024-06-03T06:19:24.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:19:24.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:19:24.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542162 2024-06-03T06:19:24.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542162 2024-06-03T06:19:24.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888069 1-167503724546 2-60129542162' 2024-06-03T06:19:24.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:19:24.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888069 2024-06-03T06:19:24.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:19:24.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:19:24.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888069 2024-06-03T06:19:24.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:19:24.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888069 2024-06-03T06:19:24.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888069' 2024-06-03T06:19:24.498 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 146028888069 2024-06-03T06:19:24.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:19:24.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888069 2024-06-03T06:19:24.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:19:25.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:19:25.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:19:26.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888069 2024-06-03T06:19:26.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:19:27.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T06:19:27.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:19:27.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888069 -lt 146028888069 2024-06-03T06:19:27.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:19:27.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724546 2024-06-03T06:19:27.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:19:27.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:19:27.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724546 2024-06-03T06:19:27.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:19:27.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-06-03T06:19:27.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2024-06-03T06:19:27.397 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 167503724546 2024-06-03T06:19:27.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:19:27.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-06-03T06:19:27.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:19:27.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542162 2024-06-03T06:19:27.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:19:27.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:19:27.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542162 2024-06-03T06:19:27.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:19:27.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542162 2024-06-03T06:19:27.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542162' 2024-06-03T06:19:27.692 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 60129542162 2024-06-03T06:19:27.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:19:28.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542162 -lt 60129542162 2024-06-03T06:19:28.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:19:28.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:19:28.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:19:28.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:19:28.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:19:28.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:19:28.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:19:28.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:19:28.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:19:28.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:19:28.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:19:28.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:19:28.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:19:28.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:19:28.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:19:29.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:19:29.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:19:29.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:19:29.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-06-03T06:19:29.152 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: { 2024-06-03T06:19:29.153 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-06-03T06:19:29.153 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:19:29.167 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: { 2024-06-03T06:19:29.167 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-06-03T06:19:29.167 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:19:29.181 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: { 2024-06-03T06:19:29.181 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-06-03T06:19:29.181 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:19:29.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-06-03T06:19:29.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-03T06:19:29.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-06-03T06:19:29.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-06-03T06:19:29.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:19:29.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-06-03T06:19:29.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-06-03T06:19:29.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:19:29.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-03T06:19:29.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-03T06:19:29.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:19:29.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-06-03T06:19:29.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-06-03T06:19:29.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:19:29.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:19:29.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:19:29.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:19:30.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-06-03T06:18:34.122774+0000 2024-06-03T06:19:30.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-06-03T06:19:30.174 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T06:19:30.174 INFO:tasks.workunit.client.0.smithi039.stdout: "deep": true, 2024-06-03T06:19:30.174 INFO:tasks.workunit.client.0.smithi039.stdout: "must": false, 2024-06-03T06:19:30.174 INFO:tasks.workunit.client.0.smithi039.stdout: "stamp": "2024-05-20T06:17:50.174637+0000" 2024-06-03T06:19:30.174 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:19:30.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-06-03T06:18:34.122774+0000 2024-06-03T06:19:30.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-03T06:19:30.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:18:34.122774+0000 2024-06-03T06:19:30.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:19:30.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:19:30.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:19:30.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:19:30.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:19:30.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:19:30.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:19:30.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:19:30.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:18:34.122774+0000 '>' 2024-06-03T06:18:34.122774+0000 2024-06-03T06:19:30.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:19:31.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:19:31.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:19:31.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:19:31.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:19:31.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:19:31.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:19:31.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:19:31.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:18:34.122774+0000 '>' 2024-06-03T06:18:34.122774+0000 2024-06-03T06:19:31.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:19:32.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:19:32.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:19:32.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:19:32.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:19:32.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:19:32.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:19:32.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:19:33.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:18:34.122774+0000 '>' 2024-06-03T06:18:34.122774+0000 2024-06-03T06:19:33.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:19:34.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:19:34.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:19:34.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:19:34.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:19:34.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:19:34.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:19:34.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:19:34.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:19:32.065256+0000 '>' 2024-06-03T06:18:34.122774+0000 2024-06-03T06:19:34.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:19:34.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-06-03T06:19:34.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:19:34.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:19:34.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:19:34.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:19:34.365 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:19:34.365 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:19:34.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:19:34.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:19:34.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:19:34.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:19:34.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:19:34.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:19:34.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:19:34.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:19:34.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:19:34.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:19:34.742 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:19:34.743 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:19:34.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:19:34.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:19:34.743 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:19:34.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888072 2024-06-03T06:19:34.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888072 2024-06-03T06:19:34.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888072' 2024-06-03T06:19:34.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:19:34.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:19:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724549 2024-06-03T06:19:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724549 2024-06-03T06:19:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888072 1-167503724549' 2024-06-03T06:19:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:19:34.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:19:35.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542165 2024-06-03T06:19:35.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542165 2024-06-03T06:19:35.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888072 1-167503724549 2-60129542165' 2024-06-03T06:19:35.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:19:35.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888072 2024-06-03T06:19:35.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:19:35.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:19:35.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888072 2024-06-03T06:19:35.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:19:35.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888072 2024-06-03T06:19:35.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888072' 2024-06-03T06:19:35.098 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 146028888072 2024-06-03T06:19:35.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:19:35.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888071 -lt 146028888072 2024-06-03T06:19:35.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:19:36.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:19:36.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:19:36.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888072 -lt 146028888072 2024-06-03T06:19:36.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:19:36.691 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724549 2024-06-03T06:19:36.691 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:19:36.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:19:36.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724549 2024-06-03T06:19:36.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:19:36.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724549 2024-06-03T06:19:36.694 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 167503724549 2024-06-03T06:19:36.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724549' 2024-06-03T06:19:36.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:19:36.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724549 -lt 167503724549 2024-06-03T06:19:36.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:19:36.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542165 2024-06-03T06:19:36.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:19:36.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:19:36.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542165 2024-06-03T06:19:36.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:19:36.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542165 2024-06-03T06:19:36.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542165' 2024-06-03T06:19:36.990 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 60129542165 2024-06-03T06:19:36.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:19:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542165 -lt 60129542165 2024-06-03T06:19:37.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:19:37.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:19:37.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:19:37.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:19:37.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:19:37.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:19:37.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:19:37.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:19:37.648 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:19:37.648 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:19:37.648 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:19:37.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:19:37.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:19:37.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:19:37.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:19:38.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:19:38.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:19:38.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:19:38.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-06-03T06:19:38.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:19:38.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:19:38.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:19:38.581 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:19:38.582 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:19:38.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:19:38.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:19:38.582 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:19:38.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888074 2024-06-03T06:19:38.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888074 2024-06-03T06:19:38.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888074' 2024-06-03T06:19:38.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:19:38.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:19:38.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724551 2024-06-03T06:19:38.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724551 2024-06-03T06:19:38.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888074 1-167503724551' 2024-06-03T06:19:38.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:19:38.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:19:38.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542167 2024-06-03T06:19:38.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542167 2024-06-03T06:19:38.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888074 1-167503724551 2-60129542167' 2024-06-03T06:19:38.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:19:38.906 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888074 2024-06-03T06:19:38.906 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:19:38.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:19:38.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888074 2024-06-03T06:19:38.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:19:38.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888074 2024-06-03T06:19:38.908 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 146028888074 2024-06-03T06:19:38.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888074' 2024-06-03T06:19:38.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:19:39.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888073 -lt 146028888074 2024-06-03T06:19:39.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:19:40.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:19:40.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:19:40.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888074 -lt 146028888074 2024-06-03T06:19:40.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:19:40.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724551 2024-06-03T06:19:40.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:19:40.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:19:40.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724551 2024-06-03T06:19:40.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:19:40.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724551 2024-06-03T06:19:40.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724551' 2024-06-03T06:19:40.501 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 167503724551 2024-06-03T06:19:40.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:19:40.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724551 -lt 167503724551 2024-06-03T06:19:40.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:19:40.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542167 2024-06-03T06:19:40.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:19:40.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:19:40.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542167 2024-06-03T06:19:40.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:19:40.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542167 2024-06-03T06:19:40.795 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 60129542167 2024-06-03T06:19:40.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542167' 2024-06-03T06:19:40.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:19:41.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542167 -lt 60129542167 2024-06-03T06:19:41.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-06-03T06:19:41.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-06-03T06:19:41.361 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:19:41.363 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-06-03T06:19:32.065342+0000 20'10 40:99 [1,0] 1 [1,0] 1 20'10 2024-06-03T06:19:32.065256+0000 20'10 2024-06-03T06:19:32.065256+0000 0 1 periodic scrub scheduled @ 2024-06-04T06:19:32.065256+0000 10 0 2024-06-03T06:19:41.364 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:19:41.364 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:19:41.364 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:19:41.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-06-03T06:19:41.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-06-03T06:19:41.642 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:19:41.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T06:19:41.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:19:41.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:19:41.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:19:41.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:19:41.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:19:41.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:19:41.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:19:41.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:19:41.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:19:41.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:19:41.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:19:41.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:19:41.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:19:41.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:19:41.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:19:41.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:19:41.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:19:41.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:19:41.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:19:41.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:19:41.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:19:41.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:19:41.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:19:41.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:19:41.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:19:41.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:19:41.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:19:41.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:19:41.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T06:19:41.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T06:19:41.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T06:19:41.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T06:19:41.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:19:41.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:19:41.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:19:41.789 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:19:41.789 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:19:41.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:19:41.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:19:41.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:19:41.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:19:41.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:19:41.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:19:41.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:19:41.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:19:41.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:19:41.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:19:41.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:19:41.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:19:41.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:19:41.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:19:41.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:19:41.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:19:41.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:19:41.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:19:41.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:19:41.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:19:41.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:19:41.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:19:41.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:19:41.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T06:19:41.798 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:19:41.798 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:19:41.798 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:19:41.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T06:19:41.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:19:41.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T06:19:41.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:19:41.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T06:19:41.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-06-03T06:19:41.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-06-03T06:19:41.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-06-03T06:19:41.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-06-03T06:19:41.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T06:19:41.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T06:19:41.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T06:19:41.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T06:19:41.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T06:19:41.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T06:19:42.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T06:19:42.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:19:42.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:19:42.178 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:19:42.178 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:19:42.179 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:19:42.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:19:42.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T06:19:42.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T06:19:42.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T06:19:42.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:19:42.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:19:42.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T06:19:42.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T06:19:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:19:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:19:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:19:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:19:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:19:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:19:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:19:42.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:19:42.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T06:19:42.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T06:19:42.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:19:42.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:19:42.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T06:19:42.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T06:19:42.457 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:19:42.457 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:19:42.457 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:19:42.457 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:19:42.457 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:19:42.457 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:19:42.457 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:19:42.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:19:42.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T06:19:42.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-06-03T06:19:42.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T06:19:42.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T06:19:42.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T06:19:42.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T06:19:42.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T06:19:42.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T06:19:43.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T06:19:43.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:19:43.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:19:43.374 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:19:43.374 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:19:43.374 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:19:43.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:19:43.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:19:43.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:19:43.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-03T06:19:43.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-06-03T06:19:43.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-06-03T06:19:43.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-03T06:19:43.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:19:43.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:19:43.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T06:19:43.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:19:43.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:19:43.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:19:43.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:19:43.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:19:43.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:19:43.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:19:43.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:19:43.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:19:43.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:19:43.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:19:43.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:19:43.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:19:43.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:19:43.411 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:19:43.411 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:19:43.411 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:19:43.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:19:43.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:19:43.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:19:43.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:19:43.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:19:43.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:19:43.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:19:43.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:19:43.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:19:43.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:19:43.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:19:43.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:19:43.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-03T06:19:43.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:19:43.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:19:43.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ea6fa3b8-13be-4aec-98ac-77e777ff1c78 2024-06-03T06:19:43.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ea6fa3b8-13be-4aec-98ac-77e777ff1c78' 2024-06-03T06:19:43.413 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 ea6fa3b8-13be-4aec-98ac-77e777ff1c78 2024-06-03T06:19:43.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:19:43.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB/YF1mYQSUGRAA6iAnni/iH3GBigNe4Ur9aA== 2024-06-03T06:19:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB/YF1mYQSUGRAA6iAnni/iH3GBigNe4Ur9aA=="}' 2024-06-03T06:19:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ea6fa3b8-13be-4aec-98ac-77e777ff1c78 -i td/osd-scrub-repair/0/new.json 2024-06-03T06:19:43.702 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:19:43.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T06:19:43.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQB/YF1mYQSUGRAA6iAnni/iH3GBigNe4Ur9aA== --osd-uuid ea6fa3b8-13be-4aec-98ac-77e777ff1c78 2024-06-03T06:19:43.745 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:43.745+0000 7f19e5a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:19:43.747 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:43.747+0000 7f19e5a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:19:43.749 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:43.749+0000 7f19e5a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:19:43.749 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:43.749+0000 7f19e5a18640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:19:43.750 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:43.749+0000 7f19e5a18640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:19:43.750 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:43.749+0000 7f19e5a18640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:19:43.828 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:43.828+0000 7f19e5a18640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T06:19:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T06:19:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:19:49.348 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:19:49.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T06:19:49.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:19:49.763 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:19:49.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T06:19:49.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-03T06:19:49.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:19:49.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:19:49.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:19:49.793 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:49.792+0000 7fad6eea2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:19:49.795 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:49.794+0000 7fad6eea2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:19:49.796 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:49.795+0000 7fad6eea2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:19:50.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T06:19:50.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:19:50.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:19:50.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:19:50.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:19:50.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:19:50.058 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:19:50.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:19:50.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:19:50.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:19:50.311 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:50.310+0000 7fad6eea2640 -1 Falling back to public interface 2024-06-03T06:19:50.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:19:51.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:19:51.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:19:51.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:19:51.351 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:19:51.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:19:51.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:19:51.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:19:52.645 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:19:52.643+0000 7fad6eea2640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:19:52.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:19:52.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:19:52.652 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:19:52.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:19:52.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:19:52.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:19:53.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:19:54.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:19:54.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:19:54.071 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:19:54.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:19:54.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:19:54.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:19:54.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:19:55.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:19:55.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:19:55.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:19:55.377 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:19:55.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:19:55.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:19:55.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:19:56.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:19:56.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:19:56.682 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:19:56.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:19:56.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:19:56.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:19:56.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:19:57.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:19:57.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:19:57.982 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:19:57.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:19:57.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:19:57.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:19:58.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:19:59.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:19:59.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:19:59.351 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:19:59.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:19:59.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:19:59.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:19:59.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:00.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:00.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:00.652 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:20:00.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:20:00.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:00.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:20:00.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:01.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:01.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:01.956 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:20:01.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:20:01.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:01.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:20:02.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:03.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:03.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:03.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:20:03.259 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:20:03.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:03.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:20:03.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:04.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:04.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:04.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:20:04.566 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:20:04.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:04.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:20:05.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:06.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:06.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:06.298 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:20:06.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:20:06.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:06.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:20:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:07.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:07.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:07.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:20:07.827 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:20:07.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:07.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:20:08.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:09.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:09.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:09.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:20:09.445 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:20:09.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:09.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:20:09.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:10.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:10.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:10.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:20:10.898 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:20:10.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:10.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:20:11.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:12.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:12.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:12.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:20:12.459 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:20:12.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:12.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:20:13.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:14.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:14.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:14.270 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:20:14.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:20:14.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:14.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:20:14.599 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1719486837,v1:127.0.0.1:6803/1719486837] [v2:127.0.0.1:6804/1719486837,v1:127.0.0.1:6805/1719486837] exists,up ea6fa3b8-13be-4aec-98ac-77e777ff1c78 2024-06-03T06:20:14.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:20:14.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:20:14.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:20:14.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-06-03T06:20:14.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-03T06:20:14.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:20:14.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:20:14.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T06:20:14.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:20:14.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:20:14.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:20:14.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:20:14.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:20:14.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:20:14.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:20:14.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:20:14.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:20:14.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:20:14.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:20:14.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:20:14.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:20:14.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:20:14.604 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:20:14.604 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:20:14.605 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:20:14.605 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:20:14.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:20:14.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:20:14.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:20:14.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:20:14.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:20:14.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:20:14.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:20:14.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:20:14.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:20:14.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:20:14.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:20:14.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-03T06:20:14.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:20:14.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:20:14.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7518623e-e489-400c-9a67-566c4e3c8304 2024-06-03T06:20:14.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7518623e-e489-400c-9a67-566c4e3c8304' 2024-06-03T06:20:14.606 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 7518623e-e489-400c-9a67-566c4e3c8304 2024-06-03T06:20:14.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:20:14.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCeYF1m1RwIJRAAX2Q8lJIjkzRkd5f0xX8Bfg== 2024-06-03T06:20:14.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCeYF1m1RwIJRAAX2Q8lJIjkzRkd5f0xX8Bfg=="}' 2024-06-03T06:20:14.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7518623e-e489-400c-9a67-566c4e3c8304 -i td/osd-scrub-repair/1/new.json 2024-06-03T06:20:15.025 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:20:15.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T06:20:15.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCeYF1m1RwIJRAAX2Q8lJIjkzRkd5f0xX8Bfg== --osd-uuid 7518623e-e489-400c-9a67-566c4e3c8304 2024-06-03T06:20:15.067 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:15.067+0000 7f7e04226640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:20:15.099 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:15.099+0000 7f7e04226640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:20:15.100 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:15.100+0000 7f7e04226640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:20:15.100 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:15.100+0000 7f7e04226640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:20:15.101 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:15.100+0000 7f7e04226640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:20:15.101 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:15.100+0000 7f7e04226640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:20:15.174 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:15.173+0000 7f7e04226640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T06:20:21.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T06:20:21.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:20:21.222 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:20:21.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T06:20:21.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:20:21.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T06:20:21.599 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:20:21.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-03T06:20:21.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:20:21.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:20:21.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:20:21.629 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:21.629+0000 7f7437ea4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:20:21.632 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:21.631+0000 7f7437ea4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:20:21.632 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:21.632+0000 7f7437ea4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:20:21.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T06:20:21.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:20:21.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:20:21.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:20:21.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:20:21.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:21.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:20:21.891 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:20:21.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:21.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:22.151 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:22.151+0000 7f7437ea4640 -1 Falling back to public interface 2024-06-03T06:20:22.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:23.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:23.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:23.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:20:23.179 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:20:23.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:23.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:23.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:24.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:24.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:24.482 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:20:24.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:20:24.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:24.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:24.539 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:24.538+0000 7f7437ea4640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:20:24.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:25.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:25.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:25.908 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:20:25.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:20:25.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:25.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:26.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:27.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:27.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:27.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:20:27.213 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:20:27.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:27.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:27.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:28.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:28.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:28.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:20:28.627 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:20:28.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:28.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:28.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:29.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:29.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:29.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:20:29.973 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:20:29.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:29.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:30.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:31.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:31.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:31.382 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:20:31.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:20:31.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:31.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:31.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:32.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:32.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:32.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:20:32.686 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:20:32.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:32.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:33.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:34.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:34.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:34.073 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:20:34.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:20:34.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:34.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:34.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:35.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:35.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:35.407 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:20:35.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:20:35.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:35.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:35.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:36.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:36.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:36.703 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:20:36.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:20:36.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:36.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:37.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:38.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:38.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:38.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:20:38.322 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:20:38.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:38.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:39.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:40.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:40.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:40.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:20:40.018 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:20:40.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:40.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:40.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:41.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:41.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:41.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:20:41.654 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:20:41.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:41.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:42.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:43.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:43.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:43.642 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:20:43.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:20:43.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:43.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:44.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:20:45.443 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:20:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:45.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:45.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:46.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:46.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:46.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:20:46.797 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:20:46.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:46.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:20:47.175 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1306889503,v1:127.0.0.1:6811/1306889503] [v2:127.0.0.1:6812/1306889503,v1:127.0.0.1:6813/1306889503] exists,up 7518623e-e489-400c-9a67-566c4e3c8304 2024-06-03T06:20:47.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:20:47.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:20:47.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:20:47.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-06-03T06:20:47.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-03T06:20:47.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:20:47.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:20:47.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T06:20:47.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:20:47.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:20:47.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:20:47.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:20:47.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:20:47.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:20:47.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:20:47.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:20:47.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:20:47.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:20:47.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:20:47.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:20:47.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:20:47.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:20:47.181 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:20:47.181 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:20:47.181 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:20:47.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:20:47.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:20:47.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:20:47.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:20:47.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:20:47.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:20:47.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:20:47.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:20:47.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:20:47.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:20:47.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:20:47.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:20:47.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-03T06:20:47.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:20:47.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:20:47.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=168f6af3-d60f-4e3b-9e18-9d92ae26554b 2024-06-03T06:20:47.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 168f6af3-d60f-4e3b-9e18-9d92ae26554b' 2024-06-03T06:20:47.183 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 168f6af3-d60f-4e3b-9e18-9d92ae26554b 2024-06-03T06:20:47.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:20:47.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/YF1mjQHXCxAA+YyAHc8qQ1zzZpKEa91RAw== 2024-06-03T06:20:47.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/YF1mjQHXCxAA+YyAHc8qQ1zzZpKEa91RAw=="}' 2024-06-03T06:20:47.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 168f6af3-d60f-4e3b-9e18-9d92ae26554b -i td/osd-scrub-repair/2/new.json 2024-06-03T06:20:47.532 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:20:47.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-03T06:20:47.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQC/YF1mjQHXCxAA+YyAHc8qQ1zzZpKEa91RAw== --osd-uuid 168f6af3-d60f-4e3b-9e18-9d92ae26554b 2024-06-03T06:20:47.577 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:47.576+0000 7f3b147ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:20:47.617 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:47.617+0000 7f3b147ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:20:47.618 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:47.618+0000 7f3b147ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:20:47.619 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:47.619+0000 7f3b147ef640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:20:47.619 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:47.619+0000 7f3b147ef640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:20:47.619 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:47.619+0000 7f3b147ef640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:20:47.692 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:47.692+0000 7f3b147ef640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T06:20:53.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-03T06:20:53.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:20:53.943 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T06:20:53.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T06:20:53.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:20:54.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T06:20:54.325 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:20:54.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-03T06:20:54.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:20:54.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:20:54.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:20:54.353 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:54.352+0000 7f6b25416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:20:54.355 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:54.354+0000 7f6b25416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:20:54.356 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:54.356+0000 7f6b25416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:20:54.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T06:20:54.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:20:54.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:20:54.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:20:54.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:20:54.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:54.615 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:20:54.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:20:54.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:54.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:20:54.875 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:54.875+0000 7f6b25416640 -1 Falling back to public interface 2024-06-03T06:20:54.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:55.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:55.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:55.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:20:55.903 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:20:55.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:55.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:20:56.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:57.196 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:20:57.195+0000 7f6b25416640 -1 osd.2 0 log_to_monitors true 2024-06-03T06:20:57.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:57.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:57.202 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:20:57.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:20:57.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:57.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:20:57.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:58.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:58.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:58.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:20:58.503 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:20:58.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:58.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:20:58.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:20:59.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:20:59.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:20:59.807 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:20:59.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:20:59.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:20:59.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:21:00.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:01.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:01.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:01.234 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:21:01.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:21:01.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:01.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:21:01.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:02.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:02.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:02.536 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:21:02.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:21:02.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:02.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:21:02.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:03.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:03.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:03.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:21:03.951 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:21:03.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:03.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:21:04.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:05.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:05.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:05.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:21:05.276 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:21:05.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:05.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:21:05.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:06.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:06.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:06.575 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:21:06.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:21:06.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:06.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:21:06.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:07.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:07.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:07.980 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:21:07.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:21:07.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:07.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:21:08.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:09.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:09.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:09.285 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:21:09.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:21:09.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:09.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:21:09.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:10.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:10.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:10.801 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:21:10.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:21:10.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:10.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:21:11.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:12.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:12.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:12.250 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:21:12.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:21:12.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:12.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:21:12.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:13.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:13.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:13.554 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:21:13.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:21:13.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:13.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:21:13.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:14.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:14.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:14.972 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:21:14.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:21:14.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:14.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:21:15.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:16.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:16.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:16.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:21:16.495 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:21:16.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:16.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:21:17.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:18.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:18.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:18.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:21:18.184 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:21:18.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:18.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:21:18.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:19.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:19.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:19.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:21:19.996 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:21:19.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:19.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:21:20.420 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1692723426,v1:127.0.0.1:6819/1692723426] [v2:127.0.0.1:6820/1692723426,v1:127.0.0.1:6821/1692723426] exists,up 168f6af3-d60f-4e3b-9e18-9d92ae26554b 2024-06-03T06:21:20.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:21:20.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:21:20.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:21:20.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-06-03T06:21:20.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-06-03T06:21:20.878 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'testpool' created 2024-06-03T06:21:20.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:21:21.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-06-03T06:21:22.438 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 2 2024-06-03T06:21:22.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-06-03T06:21:22.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:21:22.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:21:22.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:21:22.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:21:22.451 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:21:22.451 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:21:22.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:21:22.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:21:22.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:21:22.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:21:22.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:21:22.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:21:22.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:21:22.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:21:22.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:21:22.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:21:22.903 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:21:22.903 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:21:22.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:21:22.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:21:22.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:21:23.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-06-03T06:21:23.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-06-03T06:21:23.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-06-03T06:21:23.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:21:23.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:21:23.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-06-03T06:21:23.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-06-03T06:21:23.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672973' 2024-06-03T06:21:23.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:21:23.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:21:23.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-06-03T06:21:23.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-06-03T06:21:23.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672973 2-64424509448' 2024-06-03T06:21:23.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:21:23.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-06-03T06:21:23.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:21:23.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:21:23.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-06-03T06:21:23.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:21:23.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-06-03T06:21:23.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-06-03T06:21:23.486 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836497 2024-06-03T06:21:23.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:21:23.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-06-03T06:21:23.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:21:23.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-06-03T06:21:23.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:21:23.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:21:23.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-06-03T06:21:23.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:21:23.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-06-03T06:21:23.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-06-03T06:21:23.802 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672973 2024-06-03T06:21:23.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:21:24.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-06-03T06:21:24.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:21:24.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-06-03T06:21:24.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:21:24.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:21:24.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-06-03T06:21:24.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:21:24.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-06-03T06:21:24.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-06-03T06:21:24.099 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509448 2024-06-03T06:21:24.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:21:24.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-06-03T06:21:24.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:21:24.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:21:24.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:21:24.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:21:24.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:21:24.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:21:24.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:21:24.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:21:24.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:21:24.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:21:24.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:21:25.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:21:25.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:21:25.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:21:25.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:21:25.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:21:25.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:21:25.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:21:25.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-06-03T06:21:25.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-06-03T06:21:25.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-03T06:21:25.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-06-03T06:21:25.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-03T06:21:25.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:21:25.560 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-03T06:21:25.561 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:21:25.561 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:21:25.561 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:21:25.561 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:21:25.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T06:21:25.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:21:25.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T06:21:26.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-06-03T06:21:26.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:21:26.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:21:26.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:21:26.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:21:26.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-06-03T06:21:26.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:21:26.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:21:26.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:21:26.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:21:26.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:21:26.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:21:26.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:21:26.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:21:26.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:21:26.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:21:26.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:21:26.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-06-03T06:21:26.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:21:26.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:21:26.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:21:26.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:21:26.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:21:26.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-06-03T06:21:27.702 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-03T06:21:28.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-03T06:21:28.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:21:28.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:21:28.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:21:28.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:21:28.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:21:28.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:21:28.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:21:28.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:21:28.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:21:28.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:21:28.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:21:28.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:21:28.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:21:28.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:21:28.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:21:28.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:21:28.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:21:28.221 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:21:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:21:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:21:28.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:21:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:21:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:21:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:21:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:21:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:21:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:21:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:21:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:21:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:21:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-03T06:21:28.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:21:28.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:21:28.223 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:21:28.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-03T06:21:28.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:21:28.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:21:28.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:21:28.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:21:28.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:21:28.254 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:21:28.254+0000 7f781c819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:21:28.257 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:21:28.257+0000 7f781c819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:21:28.258 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:21:28.258+0000 7f781c819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:21:28.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:21:28.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:21:28.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:21:28.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:21:28.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:21:28.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:28.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:21:28.522 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:21:28.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:28.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:21:28.774 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:21:28.774+0000 7f781c819640 -1 Falling back to public interface 2024-06-03T06:21:28.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:29.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:29.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:29.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:21:29.815 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:21:29.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:29.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:21:30.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:31.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:31.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:31.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:21:31.116 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:21:31.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:31.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:21:31.126 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:21:31.125+0000 7f781c819640 -1 osd.0 21 log_to_monitors true 2024-06-03T06:21:31.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:32.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:32.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:32.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:21:32.423 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:21:32.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:32.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:21:32.730 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/739400238,v1:127.0.0.1:6803/739400238] [v2:127.0.0.1:6804/739400238,v1:127.0.0.1:6805/739400238] exists,up ea6fa3b8-13be-4aec-98ac-77e777ff1c78 2024-06-03T06:21:32.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:21:32.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:21:32.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:21:32.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:21:32.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:21:32.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:21:32.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:21:32.730 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:21:32.731 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:21:32.731 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:21:32.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:21:32.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:21:32.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:21:32.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:21:32.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:21:32.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:21:32.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:21:32.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:21:32.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:21:33.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:21:33.110 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:21:33.111 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:21:33.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:21:33.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:21:33.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:21:33.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-06-03T06:21:33.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-06-03T06:21:33.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-06-03T06:21:33.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:21:33.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:21:33.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-06-03T06:21:33.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-06-03T06:21:33.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672976' 2024-06-03T06:21:33.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:21:33.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:21:33.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-06-03T06:21:33.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-06-03T06:21:33.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672976 2-64424509451' 2024-06-03T06:21:33.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:21:33.434 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-06-03T06:21:33.434 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:21:33.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:21:33.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-06-03T06:21:33.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:21:33.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-06-03T06:21:33.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-06-03T06:21:33.437 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 107374182402 2024-06-03T06:21:33.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:21:33.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-06-03T06:21:33.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:21:33.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-06-03T06:21:33.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:21:33.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:21:33.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-06-03T06:21:33.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:21:33.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-06-03T06:21:33.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-06-03T06:21:33.761 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672976 2024-06-03T06:21:33.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:21:34.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-06-03T06:21:34.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:21:34.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-06-03T06:21:34.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:21:34.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:21:34.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-06-03T06:21:34.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:21:34.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-06-03T06:21:34.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-06-03T06:21:34.053 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509451 2024-06-03T06:21:34.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:21:34.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-06-03T06:21:34.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:21:34.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:21:34.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:21:34.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:21:34.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:21:34.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:21:34.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:21:34.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:21:34.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:21:34.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:21:34.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:21:34.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:21:34.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:21:34.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:21:34.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:21:35.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:21:35.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:21:35.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:21:35.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-06-03T06:21:35.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-03T06:21:35.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:21:35.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-06-03T06:21:35.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:21:35.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-06-03T06:21:35.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-06-03T06:21:35.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:21:35.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:21:35.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:21:35.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:21:35.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-06-03T06:21:35.953 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-06-03T06:21:35.953 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:21:35.953 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:21:35.953 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:21:35.953 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:21:36.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-06-03T06:21:20.768964+0000 2024-06-03T06:21:36.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-06-03T06:21:36.375 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T06:21:36.375 INFO:tasks.workunit.client.0.smithi039.stdout: "deep": false, 2024-06-03T06:21:36.375 INFO:tasks.workunit.client.0.smithi039.stdout: "must": false, 2024-06-03T06:21:36.375 INFO:tasks.workunit.client.0.smithi039.stdout: "stamp": "2024-05-27T06:19:56.375919+0000" 2024-06-03T06:21:36.375 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T06:21:36.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-06-03T06:21:20.768964+0000 2024-06-03T06:21:36.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-03T06:21:36.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:21:20.768964+0000 2024-06-03T06:21:36.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:21:36.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:21:36.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:21:36.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:21:36.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:21:36.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:21:36.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:21:36.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:21:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:21:20.768964+0000 '>' 2024-06-03T06:21:20.768964+0000 2024-06-03T06:21:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:21:37.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:21:37.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:21:37.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:21:37.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:21:37.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:21:37.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:21:37.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:21:37.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:21:20.768964+0000 '>' 2024-06-03T06:21:20.768964+0000 2024-06-03T06:21:37.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:21:38.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:21:38.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:21:38.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:21:38.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:21:38.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:21:38.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:21:38.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:21:39.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:21:20.768964+0000 '>' 2024-06-03T06:21:20.768964+0000 2024-06-03T06:21:39.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:21:40.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:21:40.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:21:40.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T06:21:40.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:21:40.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:21:40.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:21:40.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:21:40.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:21:37.915116+0000 '>' 2024-06-03T06:21:20.768964+0000 2024-06-03T06:21:40.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:21:40.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-06-03T06:21:40.836 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:21:40.836 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+repair 2024-06-03T06:21:38.047384+0000 21'1 26:67 [1,0] 1 [1,0] 1 21'1 2024-06-03T06:21:37.915116+0000 21'1 2024-06-03T06:21:37.915116+0000 0 1 periodic scrub scheduled @ 2024-06-04T06:21:37.915116+0000 1 0 2024-06-03T06:21:40.836 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:21:40.836 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:21:40.836 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:21:40.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-06-03T06:21:42.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-06-03T06:21:43.133 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:21:43.134 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+repair 2024-06-03T06:21:38.047384+0000 21'1 26:67 [1,0] 1 [1,0] 1 21'1 2024-06-03T06:21:37.915116+0000 21'1 2024-06-03T06:21:37.915116+0000 0 1 periodic scrub scheduled @ 2024-06-04T06:21:37.915116+0000 1 0 2024-06-03T06:21:43.134 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:21:43.134 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:21:43.134 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:21:43.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-06-03T06:21:45.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-06-03T06:21:45.431 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:21:45.431 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-03T06:21:38.097255+0000 21'1 26:68 [1,0] 1 [1,0] 1 21'1 2024-06-03T06:21:37.915116+0000 21'1 2024-06-03T06:21:37.915116+0000 0 1 periodic scrub scheduled @ 2024-06-04T06:21:37.915116+0000 1 0 2024-06-03T06:21:45.431 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:21:45.431 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:21:45.431 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:21:45.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-06-03T06:21:50.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-06-03T06:21:50.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:21:50.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:21:50.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:21:50.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:21:50.445 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:21:50.446 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:21:50.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:21:50.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:21:50.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:21:50.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:21:50.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:21:50.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:21:50.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:21:50.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:21:50.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:21:50.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:21:50.839 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:21:50.839 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:21:50.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:21:50.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:21:50.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:21:50.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182406 2024-06-03T06:21:50.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182406 2024-06-03T06:21:50.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406' 2024-06-03T06:21:50.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:21:50.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:21:51.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-06-03T06:21:51.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-06-03T06:21:51.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-42949672980' 2024-06-03T06:21:51.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:21:51.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:21:51.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-06-03T06:21:51.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-06-03T06:21:51.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-42949672980 2-64424509456' 2024-06-03T06:21:51.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:21:51.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182406 2024-06-03T06:21:51.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:21:51.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:21:51.164 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182406 2024-06-03T06:21:51.164 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:21:51.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182406 2024-06-03T06:21:51.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2024-06-03T06:21:51.166 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 107374182406 2024-06-03T06:21:51.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:21:51.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182406 2024-06-03T06:21:51.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:21:52.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:21:52.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:21:52.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182407 -lt 107374182406 2024-06-03T06:21:52.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:21:52.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-06-03T06:21:52.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:21:52.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:21:52.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-06-03T06:21:52.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:21:52.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-06-03T06:21:52.781 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672980 2024-06-03T06:21:52.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-06-03T06:21:52.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:21:53.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-06-03T06:21:53.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:21:53.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-06-03T06:21:53.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:21:53.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:21:53.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-06-03T06:21:53.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:21:53.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-06-03T06:21:53.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-06-03T06:21:53.077 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509456 2024-06-03T06:21:53.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:21:53.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-06-03T06:21:53.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:21:53.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:21:53.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:21:53.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:21:53.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:21:53.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:21:53.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:21:53.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:21:53.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:21:53.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:21:53.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:21:54.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:21:54.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:21:54.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:21:54.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:21:54.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:21:54.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:21:54.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:21:54.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-06-03T06:21:54.703 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:21:54.703 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-03T06:21:38.097255+0000 21'1 26:68 [1,0] 1 [1,0] 1 21'1 2024-06-03T06:21:37.915116+0000 21'1 2024-06-03T06:21:37.915116+0000 0 1 periodic scrub scheduled @ 2024-06-04T06:21:37.915116+0000 1 0 2024-06-03T06:21:54.703 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:21:54.703 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:21:54.703 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:21:54.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-06-03T06:21:54.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-03T06:21:54.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:21:54.715 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-03T06:21:54.715 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:21:54.715 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:21:54.715 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:21:54.715 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:21:55.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T06:21:55.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:21:55.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T06:21:55.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-03T06:21:55.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:21:55.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:21:55.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:21:55.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:21:55.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-03T06:21:55.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:21:55.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:21:55.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:21:55.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:21:55.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:21:55.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:21:55.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:21:55.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:21:55.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:21:55.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:21:55.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:21:55.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-03T06:21:55.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:21:55.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:21:55.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:21:55.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:21:55.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:21:55.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-06-03T06:21:56.810 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:21:56.810 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:21:57.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-03T06:21:57.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:21:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:21:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:21:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:21:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:21:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:21:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:21:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:21:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:21:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:21:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:21:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:21:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:21:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:21:57.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:21:57.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:21:57.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:21:57.079 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:21:57.079 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:21:57.079 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:21:57.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:21:57.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:21:57.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:21:57.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:21:57.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:21:57.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:21:57.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:21:57.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:21:57.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:21:57.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:21:57.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-03T06:21:57.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:21:57.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:21:57.081 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:21:57.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-03T06:21:57.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:21:57.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:21:57.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:21:57.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:21:57.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:21:57.112 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:21:57.112+0000 7fc018d6e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:21:57.115 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:21:57.114+0000 7fc018d6e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:21:57.116 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:21:57.116+0000 7fc018d6e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:21:57.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:21:57.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:21:57.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:21:57.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:21:57.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:21:57.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:57.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:21:57.380 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:21:57.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:57.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:21:57.634 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:21:57.634+0000 7fc018d6e640 -1 Falling back to public interface 2024-06-03T06:21:57.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:58.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:58.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:58.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:21:58.679 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:21:58.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:58.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:21:58.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:21:59.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:21:59.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:21:59.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:21:59.980 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:21:59.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:21:59.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:21:59.994 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:21:59.994+0000 7fc018d6e640 -1 osd.0 26 log_to_monitors true 2024-06-03T06:22:00.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:22:01.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:22:01.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:01.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:22:01.301 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:22:01.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:01.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:22:01.614 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/1258803494,v1:127.0.0.1:6803/1258803494] [v2:127.0.0.1:6804/1258803494,v1:127.0.0.1:6805/1258803494] exists,up ea6fa3b8-13be-4aec-98ac-77e777ff1c78 2024-06-03T06:22:01.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:22:01.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:22:01.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:22:01.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:22:01.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:22:01.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:22:01.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:22:01.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:22:01.615 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:22:01.615 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:22:01.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:22:01.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:22:01.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:22:01.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:22:01.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:22:01.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:22:01.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:22:01.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:22:01.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:22:02.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:22:02.000 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:22:02.000 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:22:02.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:22:02.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:22:02.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:22:02.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-06-03T06:22:02.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-06-03T06:22:02.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-06-03T06:22:02.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:22:02.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:22:02.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-06-03T06:22:02.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-06-03T06:22:02.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672983' 2024-06-03T06:22:02.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:22:02.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:22:02.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-06-03T06:22:02.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-06-03T06:22:02.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672983 2-64424509459' 2024-06-03T06:22:02.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:22:02.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-06-03T06:22:02.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:22:02.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:22:02.325 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-06-03T06:22:02.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:22:02.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-06-03T06:22:02.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-06-03T06:22:02.327 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 128849018882 2024-06-03T06:22:02.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:22:02.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-06-03T06:22:02.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:22:03.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:22:03.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:22:03.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-06-03T06:22:03.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:22:03.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-06-03T06:22:03.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:22:03.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:22:03.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:22:03.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-06-03T06:22:03.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-06-03T06:22:03.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-06-03T06:22:03.953 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672983 2024-06-03T06:22:03.953 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:22:04.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672983 2024-06-03T06:22:04.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:22:04.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-06-03T06:22:04.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:22:04.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:22:04.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-06-03T06:22:04.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:22:04.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-06-03T06:22:04.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-06-03T06:22:04.284 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509459 2024-06-03T06:22:04.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:22:04.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-06-03T06:22:04.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:22:04.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:22:04.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:22:04.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:22:04.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:22:04.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:22:04.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:22:04.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:22:04.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:22:04.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:22:04.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:22:05.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:22:05.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:22:05.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:22:05.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:22:05.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:22:05.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:22:05.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:22:05.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:22:05.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:22:05.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-03T06:22:05.678 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:21:36.881+0000 7f741be1d640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] planned DEEP_SCRUB_ON_ERROR] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-06-03T06:22:05.678 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:21:36.881+0000 7f741be1d640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-06-03T06:22:05.678 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:21:36.881+0000 7f741be1d640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-06-03T06:22:05.678 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:21:37.913+0000 7f741be1d640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-03T06:22:05.678 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:21:37.913+0000 7f741be1d640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-03T06:22:05.678 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:21:37.913+0000 7f741be1d640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-06-03T06:22:05.678 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:21:37.913+0000 7f741be1d640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-06-03T06:22:05.679 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:21:37.913+0000 7f741be1d640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-03T06:22:05.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-06-03T06:22:05.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-06-03T06:22:05.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-06-03T06:22:05.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-06-03T06:22:05.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T06:22:05.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:22:05.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:22:05.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:22:05.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:22:05.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:22:05.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:22:05.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:22:05.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:22:05.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:22:05.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:22:05.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:22:05.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:22:05.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:22:05.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:22:05.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:22:05.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:22:05.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:22:05.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:22:05.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:22:05.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:22:05.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:22:05.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:22:05.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:22:05.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:22:05.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:22:05.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:22:05.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:22:05.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:22:05.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T06:22:05.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T06:22:05.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T06:22:05.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T06:22:05.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:22:05.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:22:05.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:22:05.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:22:05.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:22:05.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:22:05.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:22:05.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:22:05.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:22:05.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:22:05.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:22:05.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:22:05.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:22:05.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:22:05.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:22:05.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:22:05.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:22:05.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:22:05.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:22:05.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:22:05.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:22:05.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:22:05.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:22:05.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:22:05.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:22:05.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:22:05.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:22:05.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:22:05.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T06:22:05.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:22:05.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:22:05.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:22:05.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T06:22:05.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:22:05.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T06:22:05.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:22:05.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T06:22:05.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-06-03T06:22:05.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-06-03T06:22:05.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-06-03T06:22:05.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-06-03T06:22:05.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T06:22:05.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T06:22:05.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T06:22:05.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T06:22:05.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T06:22:05.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T06:22:06.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T06:22:06.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:22:06.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:22:06.328 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:22:06.328 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:22:06.328 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:22:06.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:22:06.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T06:22:06.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T06:22:06.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T06:22:06.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:22:06.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:22:06.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T06:22:06.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T06:22:06.514 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:22:06.514 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:22:06.514 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:22:06.514 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:22:06.514 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:22:06.514 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:22:06.514 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:22:06.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:22:06.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T06:22:06.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T06:22:06.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:22:06.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:22:06.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T06:22:06.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T06:22:06.599 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:22:06.600 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:22:06.600 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:22:06.601 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:22:06.601 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:22:06.601 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:22:06.601 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:22:06.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:22:06.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T06:22:06.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-06-03T06:22:06.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T06:22:06.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T06:22:06.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T06:22:06.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T06:22:06.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T06:22:06.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T06:22:07.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T06:22:07.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:22:07.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:22:07.506 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:22:07.506 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:22:07.506 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:22:07.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:22:07.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:22:07.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:22:07.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-03T06:22:07.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-06-03T06:22:07.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-06-03T06:22:07.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-03T06:22:07.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:22:07.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:22:07.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T06:22:07.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:22:07.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:22:07.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:22:07.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:22:07.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:22:07.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:22:07.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:22:07.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:22:07.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:22:07.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:22:07.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:22:07.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:22:07.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:22:07.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:22:07.541 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:22:07.541 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:22:07.541 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:22:07.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:22:07.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:22:07.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:22:07.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:22:07.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:22:07.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:22:07.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:22:07.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:22:07.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:22:07.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:22:07.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:22:07.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:22:07.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-03T06:22:07.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:22:07.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:22:07.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=090c12ab-ea3f-4020-bfff-442e44bb9d9e 2024-06-03T06:22:07.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 090c12ab-ea3f-4020-bfff-442e44bb9d9e' 2024-06-03T06:22:07.543 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 090c12ab-ea3f-4020-bfff-442e44bb9d9e 2024-06-03T06:22:07.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:22:07.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPYV1mV+RQIRAA1UfvrwKh1FBbBuXsbFM/9w== 2024-06-03T06:22:07.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPYV1mV+RQIRAA1UfvrwKh1FBbBuXsbFM/9w=="}' 2024-06-03T06:22:07.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 090c12ab-ea3f-4020-bfff-442e44bb9d9e -i td/osd-scrub-repair/0/new.json 2024-06-03T06:22:07.779 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:22:07.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T06:22:07.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQAPYV1mV+RQIRAA1UfvrwKh1FBbBuXsbFM/9w== --osd-uuid 090c12ab-ea3f-4020-bfff-442e44bb9d9e 2024-06-03T06:22:07.822 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:07.821+0000 7f3eb2f87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:07.824 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:07.824+0000 7f3eb2f87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:07.825 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:07.825+0000 7f3eb2f87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:07.826 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:07.825+0000 7f3eb2f87640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:22:07.826 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:07.825+0000 7f3eb2f87640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:22:07.826 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:07.825+0000 7f3eb2f87640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:22:07.905 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:07.904+0000 7f3eb2f87640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T06:22:13.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T06:22:13.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:22:13.425 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:22:13.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T06:22:13.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:22:13.814 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:22:13.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T06:22:13.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-03T06:22:13.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:22:13.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:22:13.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:22:13.845 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:13.845+0000 7f2d6060e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:13.875 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:13.874+0000 7f2d6060e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:13.875 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:13.875+0000 7f2d6060e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:14.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T06:22:14.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:22:14.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:22:14.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:22:14.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:22:14.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:14.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:22:14.105 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:22:14.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:14.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:22:14.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:22:14.398 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:14.398+0000 7f2d6060e640 -1 Falling back to public interface 2024-06-03T06:22:15.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:22:15.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:15.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:22:15.393 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:22:15.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:15.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:22:15.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:22:16.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:22:16.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:16.695 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:22:16.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:22:16.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:16.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:22:16.766 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:16.765+0000 7f2d6060e640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:22:16.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:22:18.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:22:18.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:18.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:22:18.001 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:22:18.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:18.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:22:18.324 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1053468963,v1:127.0.0.1:6803/1053468963] [v2:127.0.0.1:6804/1053468963,v1:127.0.0.1:6805/1053468963] exists,up 090c12ab-ea3f-4020-bfff-442e44bb9d9e 2024-06-03T06:22:18.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:22:18.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:22:18.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:22:18.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-06-03T06:22:18.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-03T06:22:18.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:22:18.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:22:18.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T06:22:18.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:22:18.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:22:18.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:22:18.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:22:18.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:22:18.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:22:18.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:22:18.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:22:18.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:22:18.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:22:18.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:22:18.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:22:18.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:22:18.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:22:18.330 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:22:18.330 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:22:18.330 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:22:18.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:22:18.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:22:18.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:22:18.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:22:18.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:22:18.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:22:18.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:22:18.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:22:18.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:22:18.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:22:18.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:22:18.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:22:18.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-03T06:22:18.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:22:18.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:22:18.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a7814e1e-9ec4-4049-9203-5df8ba57cc0b 2024-06-03T06:22:18.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a7814e1e-9ec4-4049-9203-5df8ba57cc0b' 2024-06-03T06:22:18.332 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 a7814e1e-9ec4-4049-9203-5df8ba57cc0b 2024-06-03T06:22:18.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:22:18.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAaYV1mVTqwFBAAeY9GJPLZPN5XNHdqcjFbqQ== 2024-06-03T06:22:18.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAaYV1mVTqwFBAAeY9GJPLZPN5XNHdqcjFbqQ=="}' 2024-06-03T06:22:18.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a7814e1e-9ec4-4049-9203-5df8ba57cc0b -i td/osd-scrub-repair/1/new.json 2024-06-03T06:22:18.727 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:22:18.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T06:22:18.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQAaYV1mVTqwFBAAeY9GJPLZPN5XNHdqcjFbqQ== --osd-uuid a7814e1e-9ec4-4049-9203-5df8ba57cc0b 2024-06-03T06:22:18.771 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:18.771+0000 7f9d9f52f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:18.820 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:18.819+0000 7f9d9f52f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:18.821 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:18.820+0000 7f9d9f52f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:18.821 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:18.821+0000 7f9d9f52f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:22:18.821 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:18.821+0000 7f9d9f52f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:22:18.821 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:18.821+0000 7f9d9f52f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:22:18.895 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:18.895+0000 7f9d9f52f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T06:22:24.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T06:22:24.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:22:24.689 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:22:24.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T06:22:24.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:22:25.062 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:22:25.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T06:22:25.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-03T06:22:25.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:22:25.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:22:25.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:22:25.092 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:25.091+0000 7fab3d140640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:25.094 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:25.093+0000 7fab3d140640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:25.095 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:25.094+0000 7fab3d140640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:25.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T06:22:25.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:22:25.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:22:25.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:22:25.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:22:25.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:25.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:22:25.356 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:22:25.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:25.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:22:25.624 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:25.623+0000 7fab3d140640 -1 Falling back to public interface 2024-06-03T06:22:25.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:22:26.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:22:26.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:26.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:22:26.648 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:22:26.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:26.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:22:26.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:22:27.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:22:27.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:27.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:22:27.945 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:22:27.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:27.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:22:28.043 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:28.042+0000 7fab3d140640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:22:28.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:22:29.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:22:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:29.252 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:22:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:22:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:29.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:22:29.563 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4108958465,v1:127.0.0.1:6811/4108958465] [v2:127.0.0.1:6812/4108958465,v1:127.0.0.1:6813/4108958465] exists,up a7814e1e-9ec4-4049-9203-5df8ba57cc0b 2024-06-03T06:22:29.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:22:29.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:22:29.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:22:29.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-06-03T06:22:29.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-03T06:22:29.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:22:29.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:22:29.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T06:22:29.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:22:29.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:22:29.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:22:29.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:22:29.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:22:29.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:22:29.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:22:29.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:22:29.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:22:29.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:22:29.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:22:29.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:22:29.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:22:29.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:22:29.568 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:22:29.568 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:22:29.569 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:22:29.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:22:29.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:22:29.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:22:29.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:22:29.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:22:29.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:22:29.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:22:29.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:22:29.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:22:29.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:22:29.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:22:29.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:22:29.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-03T06:22:29.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:22:29.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:22:29.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4576b14f-1bb2-4b4d-9087-e20d1686e242 2024-06-03T06:22:29.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4576b14f-1bb2-4b4d-9087-e20d1686e242' 2024-06-03T06:22:29.570 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 4576b14f-1bb2-4b4d-9087-e20d1686e242 2024-06-03T06:22:29.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:22:29.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlYV1mNYD3IhAAZVr2nkmO9j9CD03g5CmPaw== 2024-06-03T06:22:29.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlYV1mNYD3IhAAZVr2nkmO9j9CD03g5CmPaw=="}' 2024-06-03T06:22:29.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4576b14f-1bb2-4b4d-9087-e20d1686e242 -i td/osd-scrub-repair/2/new.json 2024-06-03T06:22:29.978 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:22:29.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-03T06:22:29.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQAlYV1mNYD3IhAAZVr2nkmO9j9CD03g5CmPaw== --osd-uuid 4576b14f-1bb2-4b4d-9087-e20d1686e242 2024-06-03T06:22:30.024 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:30.023+0000 7fdf6de11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:30.063 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:30.062+0000 7fdf6de11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:30.064 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:30.063+0000 7fdf6de11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:30.065 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:30.063+0000 7fdf6de11640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:22:30.065 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:30.063+0000 7fdf6de11640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:22:30.065 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:30.063+0000 7fdf6de11640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:22:30.138 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:30.137+0000 7fdf6de11640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T06:22:36.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-03T06:22:36.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:22:36.095 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T06:22:36.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T06:22:36.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:22:36.474 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:22:36.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T06:22:36.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-03T06:22:36.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:22:36.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:22:36.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:22:36.504 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:36.504+0000 7f286c21d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:36.506 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:36.506+0000 7f286c21d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:36.507 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:36.507+0000 7f286c21d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:36.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T06:22:36.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:22:36.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:22:36.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:22:36.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:22:36.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:36.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:22:36.779 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:22:36.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:36.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:22:37.030 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:37.030+0000 7f286c21d640 -1 Falling back to public interface 2024-06-03T06:22:37.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:22:38.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:22:38.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:38.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:22:38.066 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:22:38.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:38.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:22:38.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:22:39.322 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:39.322+0000 7f286c21d640 -1 osd.2 0 log_to_monitors true 2024-06-03T06:22:39.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:22:39.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:39.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:22:39.367 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:22:39.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:39.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:22:39.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:22:40.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:22:40.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:40.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:22:40.739 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:22:40.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:40.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:22:41.039 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2147707625,v1:127.0.0.1:6819/2147707625] [v2:127.0.0.1:6820/2147707625,v1:127.0.0.1:6821/2147707625] exists,up 4576b14f-1bb2-4b4d-9087-e20d1686e242 2024-06-03T06:22:41.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:22:41.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:22:41.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:22:41.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-06-03T06:22:41.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-06-03T06:22:41.424 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'testpool' created 2024-06-03T06:22:41.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:22:42.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-06-03T06:22:42.956 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 2 2024-06-03T06:22:42.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-06-03T06:22:42.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:22:42.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:22:42.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:22:42.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:22:42.969 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:22:42.969 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:22:42.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:22:42.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:22:42.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:22:43.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:22:43.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:22:43.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:22:43.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:22:43.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:22:43.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:22:43.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:22:43.454 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:22:43.454 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:22:43.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:22:43.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:22:43.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:22:43.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-06-03T06:22:43.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-06-03T06:22:43.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-06-03T06:22:43.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:22:43.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:22:43.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-06-03T06:22:43.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-06-03T06:22:43.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2024-06-03T06:22:43.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:22:43.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:22:43.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-06-03T06:22:43.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-06-03T06:22:43.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509442' 2024-06-03T06:22:43.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:22:43.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-06-03T06:22:43.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:22:43.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:22:43.875 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-06-03T06:22:43.875 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:22:43.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-06-03T06:22:43.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-06-03T06:22:43.877 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836487 2024-06-03T06:22:43.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:22:44.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836487 2024-06-03T06:22:44.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:22:44.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-06-03T06:22:44.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:22:44.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:22:44.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-06-03T06:22:44.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:22:44.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-06-03T06:22:44.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-06-03T06:22:44.201 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672964 2024-06-03T06:22:44.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:22:44.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672963 -lt 42949672964 2024-06-03T06:22:44.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:22:45.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:22:45.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:22:45.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672964 2024-06-03T06:22:45.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:22:45.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-06-03T06:22:45.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:22:45.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:22:45.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-06-03T06:22:45.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:22:45.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-06-03T06:22:45.797 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509442 2024-06-03T06:22:45.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-06-03T06:22:45.798 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:22:46.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-06-03T06:22:46.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:22:46.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:22:46.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:22:46.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:22:46.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:22:46.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:22:46.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:22:46.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:22:46.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:22:46.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:22:46.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:22:46.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:22:46.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:22:46.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:22:46.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:22:47.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:22:47.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:22:47.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:22:47.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-06-03T06:22:47.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-06-03T06:22:47.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-03T06:22:47.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-06-03T06:22:47.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-03T06:22:47.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:22:47.247 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-03T06:22:47.247 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:22:47.247 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:22:47.247 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:22:47.247 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:22:47.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T06:22:47.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:22:47.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T06:22:47.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-06-03T06:22:47.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:22:47.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:22:47.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:22:47.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:22:47.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-06-03T06:22:47.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:22:47.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:22:47.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:22:47.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:22:47.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:22:47.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:22:47.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:22:47.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:22:47.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:22:47.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:22:48.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:22:48.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-06-03T06:22:48.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:22:48.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:22:48.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:22:48.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:22:48.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:22:48.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-06-03T06:22:49.406 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-03T06:22:49.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-03T06:22:49.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:22:49.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:22:49.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:22:49.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:22:49.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:22:49.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:22:49.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:22:49.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:22:49.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:22:49.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:22:49.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:22:49.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:22:49.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:22:49.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:22:49.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:22:49.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:22:49.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:22:49.925 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:22:49.925 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:22:49.925 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:22:49.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:22:49.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:22:49.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:22:49.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:22:49.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:22:49.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:22:49.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:22:49.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:22:49.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:22:49.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:22:49.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-03T06:22:49.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:22:49.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:22:49.927 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:22:49.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-03T06:22:49.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:22:49.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:22:49.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:22:49.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:22:49.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:22:49.958 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:49.958+0000 7fa73b641640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:49.961 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:49.960+0000 7fa73b641640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:49.962 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:49.962+0000 7fa73b641640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:22:50.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:22:50.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:22:50.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:22:50.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:22:50.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:22:50.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:50.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:22:50.224 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:22:50.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:50.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:22:50.477 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:50.477+0000 7fa73b641640 -1 Falling back to public interface 2024-06-03T06:22:50.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:22:51.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:22:51.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:51.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:22:51.511 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:22:51.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:51.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:22:51.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:22:52.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:22:52.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:52.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:22:52.811 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:22:52.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:52.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:22:52.894 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:22:52.893+0000 7fa73b641640 -1 osd.0 20 log_to_monitors true 2024-06-03T06:22:53.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:22:54.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:22:54.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:22:54.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:22:54.115 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:22:54.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:22:54.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:22:54.416 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/299676508,v1:127.0.0.1:6803/299676508] [v2:127.0.0.1:6804/299676508,v1:127.0.0.1:6805/299676508] exists,up 090c12ab-ea3f-4020-bfff-442e44bb9d9e 2024-06-03T06:22:54.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:22:54.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:22:54.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:22:54.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:22:54.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:22:54.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:22:54.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:22:54.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:22:54.417 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:22:54.417 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:22:54.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:22:54.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:22:54.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:22:54.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:22:54.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:22:54.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:22:54.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:22:54.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:22:54.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:22:54.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:22:54.797 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:22:54.797 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:22:54.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:22:54.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:22:54.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:22:54.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215106 2024-06-03T06:22:54.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215106 2024-06-03T06:22:54.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106' 2024-06-03T06:22:54.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:22:54.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:22:55.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-06-03T06:22:55.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-06-03T06:22:55.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672968' 2024-06-03T06:22:55.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:22:55.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:22:55.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-06-03T06:22:55.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-06-03T06:22:55.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672968 2-64424509445' 2024-06-03T06:22:55.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:22:55.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215106 2024-06-03T06:22:55.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:22:55.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:22:55.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215106 2024-06-03T06:22:55.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:22:55.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215106 2024-06-03T06:22:55.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-06-03T06:22:55.158 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 103079215106 2024-06-03T06:22:55.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:22:55.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 103079215106 2024-06-03T06:22:55.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:22:56.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:22:56.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:22:56.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215106 -lt 103079215106 2024-06-03T06:22:56.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:22:56.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-06-03T06:22:56.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:22:56.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:22:56.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-06-03T06:22:56.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:22:56.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-06-03T06:22:56.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-06-03T06:22:56.752 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672968 2024-06-03T06:22:56.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:22:57.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672968 2024-06-03T06:22:57.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:22:57.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-06-03T06:22:57.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:22:57.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:22:57.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-06-03T06:22:57.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:22:57.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-06-03T06:22:57.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-06-03T06:22:57.060 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509445 2024-06-03T06:22:57.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:22:57.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-06-03T06:22:57.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:22:57.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:22:57.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:22:57.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:22:57.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:22:57.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:22:57.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:22:57.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:22:57.720 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:22:57.720 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:22:57.720 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:22:58.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:22:58.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:22:58.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:22:58.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:22:58.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:22:58.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:22:58.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:22:58.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-06-03T06:22:58.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-03T06:22:58.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:22:58.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-06-03T06:22:58.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:22:58.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-06-03T06:22:58.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-06-03T06:22:58.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:22:58.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:22:58.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:22:58.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:22:58.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-06-03T06:22:58.954 INFO:tasks.workunit.client.0.smithi039.stdout:Affected PG 1.0 w/ primary 1 2024-06-03T06:22:58.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-06-03T06:22:58.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-06-03T06:22:58.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:22:58.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:22:58.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:22:58.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:22:59.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-06-03T06:22:41.375898+0000 2024-06-03T06:22:59.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-06-03T06:22:59.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-06-03T06:22:59.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-06-03T06:22:59.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-06-03T06:22:59.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T06:22:59.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:22:59.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:22:59.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T06:22:59.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-06-03T06:22:41.375898+0000 2024-06-03T06:22:59.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-06-03T06:22:59.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-03T06:22:59.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-03T06:22:59.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-06-03T06:22:59.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=3.0 2024-06-03T06:22:59.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-03T06:22:59.572 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-03T06:22:59.572 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-03T06:22:59.572 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-03T06:22:59.572 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:22:59.572 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:22:59.572 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:22:59.572 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.1.asok 2024-06-03T06:22:59.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69989/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-06-03T06:22:59.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-03T06:22:59.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-06-03T06:22:59.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-03T06:22:59.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-03T06:22:59.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_auto_repair 2024-06-03T06:22:59.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-03T06:22:59.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-03T06:22:59.665 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-03T06:22:59.665 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-03T06:22:59.665 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-03T06:22:59.665 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:22:59.665 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:22:59.665 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:22:59.665 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.1.asok 2024-06-03T06:22:59.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69989/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-06-03T06:22:59.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-03T06:22:59.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-06-03T06:22:59.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:22:59.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:23:00.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:23:00.045 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:23:00.045 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:23:00.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:23:00.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:23:00.045 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:23:00.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215108 2024-06-03T06:23:00.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215108 2024-06-03T06:23:00.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215108' 2024-06-03T06:23:00.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:23:00.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:23:00.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-06-03T06:23:00.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-06-03T06:23:00.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215108 1-42949672970' 2024-06-03T06:23:00.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:23:00.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:23:00.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-06-03T06:23:00.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-06-03T06:23:00.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215108 1-42949672970 2-64424509447' 2024-06-03T06:23:00.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:23:00.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215108 2024-06-03T06:23:00.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:23:00.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:23:00.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215108 2024-06-03T06:23:00.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:23:00.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215108 2024-06-03T06:23:00.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215108' 2024-06-03T06:23:00.371 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 103079215108 2024-06-03T06:23:00.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:23:00.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215107 -lt 103079215108 2024-06-03T06:23:00.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:23:01.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:23:01.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:23:02.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215108 -lt 103079215108 2024-06-03T06:23:02.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:23:02.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-06-03T06:23:02.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:23:02.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:23:02.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-06-03T06:23:02.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:23:02.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-06-03T06:23:02.012 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672970 2024-06-03T06:23:02.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-06-03T06:23:02.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:23:02.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-06-03T06:23:02.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:23:02.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-06-03T06:23:02.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:23:02.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:23:02.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-06-03T06:23:02.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:23:02.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-06-03T06:23:02.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-06-03T06:23:02.304 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509447 2024-06-03T06:23:02.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:23:02.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509447 2024-06-03T06:23:02.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-06-03T06:23:02.592 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03 06:23:02.593331300+00:00 2024-06-03T06:23:02.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-06-03T06:23:02.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-03T06:23:02.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-03T06:23:02.594 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:23:02.594 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:23:02.594 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:23:02.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.1.asok 2024-06-03T06:23:02.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69989/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-06-03T06:23:02.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-06-03T06:23:02.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-03T06:23:02.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-03T06:23:02.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-03T06:23:02.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-03T06:23:02.799 INFO:tasks.workunit.client.0.smithi039.stdout:{"deep":true,"must":false,"stamp":"2024-05-20T06:21:22.680553+0000"}0 ) state now: "active+clean" 2024-06-03T06:23:02.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-06-03T06:23:02.799 INFO:tasks.workunit.client.0.smithi039.stdout:"active+clean" 2024-06-03T06:23:02.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-03T06:23:02.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-03T06:23:02.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-06-03T06:23:02.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-03T06:23:03.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-03T06:23:03.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-03T06:23:03.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-03T06:23:03.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-03T06:23:03.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-03T06:23:03.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-06-03T06:23:03.212 INFO:tasks.workunit.client.0.smithi039.stdout:1 ) state now: "active+clean" 2024-06-03T06:23:03.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-03T06:23:03.212 INFO:tasks.workunit.client.0.smithi039.stdout:"active+clean" 2024-06-03T06:23:03.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-03T06:23:03.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-06-03T06:23:03.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-03T06:23:03.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-03T06:23:03.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-03T06:23:03.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-03T06:23:03.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-03T06:23:03.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-03T06:23:03.625 INFO:tasks.workunit.client.0.smithi039.stdout:2 ) state now: "active+clean" 2024-06-03T06:23:03.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2024-06-03T06:23:03.626 INFO:tasks.workunit.client.0.smithi039.stdout:"active+clean" 2024-06-03T06:23:03.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-03T06:23:03.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-03T06:23:03.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-06-03T06:23:03.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-03T06:23:03.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-03T06:23:03.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-03T06:23:03.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-03T06:23:03.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-03T06:23:04.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-03T06:23:04.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean"' 2024-06-03T06:23:04.041 INFO:tasks.workunit.client.0.smithi039.stdout:3 ) state now: "active+clean" 2024-06-03T06:23:04.041 INFO:tasks.workunit.client.0.smithi039.stdout:"active+clean" 2024-06-03T06:23:04.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-03T06:23:04.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-03T06:23:04.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 3 == 4 ']' 2024-06-03T06:23:04.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-03T06:23:04.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-03T06:23:04.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-03T06:23:04.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-03T06:23:04.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-03T06:23:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-03T06:23:04.455 INFO:tasks.workunit.client.0.smithi039.stdout:4 ) state now: "active+clean" 2024-06-03T06:23:04.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 4 ') state now: ' '"active+clean"' 2024-06-03T06:23:04.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-03T06:23:04.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-03T06:23:04.456 INFO:tasks.workunit.client.0.smithi039.stdout:"active+clean" 2024-06-03T06:23:04.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 4 == 4 ']' 2024-06-03T06:23:04.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo 'loop --------> ' 4 2024-06-03T06:23:04.458 INFO:tasks.workunit.client.0.smithi039.stdout:loop --------> 4 2024-06-03T06:23:04.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-03T06:23:04.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-03T06:23:04.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-03T06:23:04.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-03T06:23:04.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-03T06:23:04.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-03T06:23:04.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 5 ') state now: ' '"active+clean"' 2024-06-03T06:23:04.868 INFO:tasks.workunit.client.0.smithi039.stdout:5 ) state now: "active+clean" 2024-06-03T06:23:04.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-03T06:23:04.868 INFO:tasks.workunit.client.0.smithi039.stdout:"active+clean" 2024-06-03T06:23:04.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-03T06:23:04.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 5 == 4 ']' 2024-06-03T06:23:04.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-03T06:23:05.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-03T06:23:05.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-03T06:23:05.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-03T06:23:05.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-03T06:23:05.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-06-03T06:23:05.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 6 ') state now: ' '"active+clean+scrubbing+deep"' 2024-06-03T06:23:05.282 INFO:tasks.workunit.client.0.smithi039.stdout:6 ) state now: "active+clean+scrubbing+deep" 2024-06-03T06:23:05.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-03T06:23:05.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-06-03T06:23:05.283 INFO:tasks.workunit.client.0.smithi039.stdout:found scrub 2024-06-03T06:23:05.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-06-03T06:23:05.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-06-03T06:23:05.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-06-03T06:23:05.284 INFO:tasks.workunit.client.0.smithi039.stdout:initiate_and_fetch_state ret: 0 2024-06-03T06:23:05.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-06-03T06:23:05.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-03T06:23:05.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=td/osd-scrub-repair 2024-06-03T06:23:05.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-06-03T06:23:05.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=0 2024-06-03T06:23:05.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-03T06:23:05.285 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.td/osd-scrub-repair 2024-06-03T06:23:05.286 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.td/osd-scrub-repair 2024-06-03T06:23:05.286 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-06-03T06:23:05.286 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:23:05.286 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:23:05.286 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:23:05.286 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.td/osd-scrub-repair.asok 2024-06-03T06:23:05.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69989/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-06-03T06:23:05.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test == true 2024-06-03T06:23:05.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1168: test: ==: unary operator expected 2024-06-03T06:23:05.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-06-03T06:23:05.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-06-03T06:23:05.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-06-03T06:23:05.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-06-03T06:23:05.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-06-03T06:23:05.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-03T06:23:05.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-03T06:23:05.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-03T06:23:05.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-06-03T06:23:05.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-06-03T06:23:05.465 INFO:tasks.workunit.client.0.smithi039.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-06-03T06:23:05.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-06-03T06:23:05.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-06-03T06:23:05.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-06-03T06:23:05.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-06-03T06:23:05.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-03T06:23:05.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-03T06:23:05.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-03T06:23:05.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-06-03T06:23:05.881 INFO:tasks.workunit.client.0.smithi039.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-06-03T06:23:05.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-06-03T06:23:05.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-06-03T06:23:05.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-06-03T06:23:05.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-06-03T06:23:06.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-06-03T06:23:06.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-03T06:23:06.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-03T06:23:06.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-03T06:23:06.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-06-03T06:23:06.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-06-03T06:23:06.295 INFO:tasks.workunit.client.0.smithi039.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-06-03T06:23:06.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-06-03T06:23:06.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-06-03T06:23:06.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-06-03T06:23:06.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-06-03T06:23:06.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-03T06:23:06.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-03T06:23:06.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-03T06:23:06.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-06-03T06:23:06.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-06-03T06:23:06.710 INFO:tasks.workunit.client.0.smithi039.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-06-03T06:23:06.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-06-03T06:23:06.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-06-03T06:23:06.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-06-03T06:23:07.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-06-03T06:23:07.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-03T06:23:07.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-03T06:23:07.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-03T06:23:07.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-06-03T06:23:07.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-06-03T06:23:07.124 INFO:tasks.workunit.client.0.smithi039.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-06-03T06:23:07.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-06-03T06:23:07.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-06-03T06:23:07.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-06-03T06:23:07.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:23:07.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:23:07.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:23:07.411 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:23:07.411 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:23:07.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:23:07.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:23:07.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:23:07.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215110 2024-06-03T06:23:07.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215110 2024-06-03T06:23:07.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110' 2024-06-03T06:23:07.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:23:07.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:23:07.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-06-03T06:23:07.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-06-03T06:23:07.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110 1-42949672972' 2024-06-03T06:23:07.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:23:07.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:23:07.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-06-03T06:23:07.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-06-03T06:23:07.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110 1-42949672972 2-64424509450' 2024-06-03T06:23:07.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:23:07.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215110 2024-06-03T06:23:07.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:23:07.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:23:07.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215110 2024-06-03T06:23:07.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:23:07.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215110 2024-06-03T06:23:07.775 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 103079215110 2024-06-03T06:23:07.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215110' 2024-06-03T06:23:07.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:23:08.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215110 -lt 103079215110 2024-06-03T06:23:08.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:23:08.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-06-03T06:23:08.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:23:08.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:23:08.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-06-03T06:23:08.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:23:08.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-06-03T06:23:08.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-06-03T06:23:08.066 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672972 2024-06-03T06:23:08.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:23:08.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672972 2024-06-03T06:23:08.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:23:09.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:23:09.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:23:09.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672972 2024-06-03T06:23:09.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:23:10.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T06:23:10.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:23:10.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672972 2024-06-03T06:23:10.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:23:10.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-06-03T06:23:10.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:23:10.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:23:10.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-06-03T06:23:10.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:23:10.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-06-03T06:23:10.963 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509450 2024-06-03T06:23:10.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-06-03T06:23:10.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:23:11.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-06-03T06:23:11.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-06-03T06:23:11.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-06-03T06:23:11.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-03T06:23:11.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-03T06:23:11.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-03T06:23:11.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-06-03T06:23:11.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-06-03T06:23:11.666 INFO:tasks.workunit.client.0.smithi039.stdout:wait-scrub-end state now: "active+clean" 2024-06-03T06:23:11.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-06-03T06:23:11.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-06-03T06:23:11.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-06-03T06:23:11.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-06-03T06:23:11.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-06-03T06:23:11.939 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T06:23:11.939 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-03T06:23:08.512224+0000 20'1 25:55 [1,0] 1 [1,0] 1 20'1 2024-06-03T06:23:08.137656+0000 20'1 2024-06-03T06:23:08.137656+0000 0 4 periodic scrub scheduled @ 2024-06-04T06:23:08.137656+0000 1 0 2024-06-03T06:23:11.939 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T06:23:11.939 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T06:23:11.939 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T06:23:11.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-06-03T06:23:11.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-03T06:23:11.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:23:11.951 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-03T06:23:11.951 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:23:11.951 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:23:11.951 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:23:11.951 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:23:12.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T06:23:12.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:23:12.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T06:23:12.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-03T06:23:12.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:23:12.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:23:12.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:23:12.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:23:12.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-03T06:23:12.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:23:12.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:23:12.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:23:12.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:23:12.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:23:12.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:23:12.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:23:12.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:23:12.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:23:12.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:23:12.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:23:12.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-03T06:23:12.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:23:12.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:23:12.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:23:12.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:23:12.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:23:12.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-06-03T06:23:14.069 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:23:14.069 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:23:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-03T06:23:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:23:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:23:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:23:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:23:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:23:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:23:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:23:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:23:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:23:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:23:14.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:23:14.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:23:14.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:23:14.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:23:14.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:23:14.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:23:14.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:23:14.338 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:23:14.338 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:23:14.338 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:23:14.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:23:14.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:23:14.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:23:14.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:23:14.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:23:14.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:23:14.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:23:14.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:23:14.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:23:14.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:23:14.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-03T06:23:14.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:23:14.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:23:14.341 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:23:14.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-03T06:23:14.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:23:14.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:23:14.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:23:14.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:23:14.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:23:14.372 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:14.371+0000 7f725220c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:23:14.374 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:14.374+0000 7f725220c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:23:14.375 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:14.375+0000 7f725220c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:23:14.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:23:14.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:23:14.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:23:14.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:23:14.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:23:14.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:14.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:23:14.638 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:23:14.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:14.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:14.894 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:14.894+0000 7f725220c640 -1 Falling back to public interface 2024-06-03T06:23:14.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:15.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:15.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:15.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:23:15.926 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:23:15.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:15.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:16.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:17.157 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:17.155+0000 7f725220c640 -1 osd.0 25 log_to_monitors true 2024-06-03T06:23:17.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:17.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:17.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:23:17.228 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:23:17.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:17.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:17.575 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/3183924853,v1:127.0.0.1:6803/3183924853] [v2:127.0.0.1:6804/3183924853,v1:127.0.0.1:6805/3183924853] exists,up 090c12ab-ea3f-4020-bfff-442e44bb9d9e 2024-06-03T06:23:17.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:23:17.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:23:17.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:23:17.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:23:17.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:23:17.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:23:17.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:23:17.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:23:17.576 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:23:17.577 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:23:17.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:23:17.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:23:17.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:23:17.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:23:17.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:23:17.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:23:17.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:23:17.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:23:17.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:23:17.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:23:17.956 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:23:17.956 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:23:17.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:23:17.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:23:17.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:23:18.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-06-03T06:23:18.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-06-03T06:23:18.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-06-03T06:23:18.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:23:18.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:23:18.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-06-03T06:23:18.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-06-03T06:23:18.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672975' 2024-06-03T06:23:18.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:23:18.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:23:18.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-06-03T06:23:18.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-06-03T06:23:18.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672975 2-64424509453' 2024-06-03T06:23:18.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:23:18.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-06-03T06:23:18.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:23:18.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:23:18.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-06-03T06:23:18.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:23:18.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-06-03T06:23:18.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-06-03T06:23:18.303 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 124554051586 2024-06-03T06:23:18.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:23:18.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-06-03T06:23:18.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:23:19.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:23:19.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:23:19.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-06-03T06:23:19.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:23:19.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-06-03T06:23:19.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:23:19.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:23:19.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-06-03T06:23:19.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:23:19.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-06-03T06:23:19.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-06-03T06:23:19.949 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672975 2024-06-03T06:23:19.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:23:20.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672975 2024-06-03T06:23:20.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:23:20.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-06-03T06:23:20.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:23:20.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:23:20.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-06-03T06:23:20.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:23:20.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-06-03T06:23:20.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-06-03T06:23:20.269 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509453 2024-06-03T06:23:20.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:23:20.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-06-03T06:23:20.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:23:20.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:23:20.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:23:20.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:23:20.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:23:20.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:23:20.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:23:20.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:23:20.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:23:20.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:23:20.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:23:21.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:23:21.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:23:21.254 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:23:21.254 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:23:21.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:23:21.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:23:21.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:23:21.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-06-03T06:23:21.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-03T06:23:21.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:23:21.631 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-03T06:23:21.632 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T06:23:21.632 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:23:21.632 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:23:21.632 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:23:21.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T06:23:21.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-03T06:23:21.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T06:23:22.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-03T06:23:22.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:23:22.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:23:22.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:23:22.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:23:22.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-03T06:23:22.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:23:22.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:23:22.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:23:22.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:23:22.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:23:22.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:23:22.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:23:22.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:23:22.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:23:22.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:23:22.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:23:22.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-03T06:23:22.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:23:22.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:23:22.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:23:22.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:23:22.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:23:22.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-03T06:23:24.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-03T06:23:24.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:23:24.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:23:24.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:23:24.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:23:24.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:23:24.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:23:24.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:23:24.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:23:24.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:23:24.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:23:24.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:23:24.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:23:24.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:23:24.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:23:24.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:23:24.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:23:24.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:23:24.060 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:23:24.060 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:23:24.060 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:23:24.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:23:24.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:23:24.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:23:24.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:23:24.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:23:24.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:23:24.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:23:24.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:23:24.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:23:24.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:23:24.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-03T06:23:24.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:23:24.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:23:24.062 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:23:24.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-03T06:23:24.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:23:24.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:23:24.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:23:24.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:23:24.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:23:24.093 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:24.092+0000 7fe35fe1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:23:24.095 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:24.095+0000 7fe35fe1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:23:24.096 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:24.096+0000 7fe35fe1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:23:24.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:23:24.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:23:24.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:23:24.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:23:24.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:23:24.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:24.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:23:24.360 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:23:24.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:24.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:24.611 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:24.611+0000 7fe35fe1d640 -1 Falling back to public interface 2024-06-03T06:23:24.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:25.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:25.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:25.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:23:25.651 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:23:25.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:25.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:25.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:26.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:26.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:26.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:23:26.959 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:23:26.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:26.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:27.002 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:27.002+0000 7fe35fe1d640 -1 osd.0 30 log_to_monitors true 2024-06-03T06:23:27.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:28.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:28.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:28.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:23:28.335 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:23:28.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:28.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:28.640 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 34 up_thru 0 down_at 31 last_clean_interval [29,30) [v2:127.0.0.1:6802/4289739863,v1:127.0.0.1:6803/4289739863] [v2:127.0.0.1:6804/4289739863,v1:127.0.0.1:6805/4289739863] exists,up 090c12ab-ea3f-4020-bfff-442e44bb9d9e 2024-06-03T06:23:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:23:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:23:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:23:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:23:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:23:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:23:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:23:28.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:23:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:23:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:23:28.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:23:28.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:23:28.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:23:28.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:23:28.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:23:28.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:23:28.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:23:28.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:23:28.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:23:29.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:23:29.046 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:23:29.046 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:23:29.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:23:29.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:23:29.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:23:29.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-06-03T06:23:29.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-06-03T06:23:29.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066' 2024-06-03T06:23:29.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:23:29.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:23:29.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-06-03T06:23:29.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-06-03T06:23:29.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-42949672978' 2024-06-03T06:23:29.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:23:29.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:23:29.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-06-03T06:23:29.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-06-03T06:23:29.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-42949672978 2-64424509456' 2024-06-03T06:23:29.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:23:29.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888066 2024-06-03T06:23:29.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:23:29.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:23:29.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888066 2024-06-03T06:23:29.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:23:29.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-06-03T06:23:29.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-06-03T06:23:29.373 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 146028888066 2024-06-03T06:23:29.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:23:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 146028888066 2024-06-03T06:23:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:23:30.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:23:30.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:23:30.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888066 -lt 146028888066 2024-06-03T06:23:30.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:23:30.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-06-03T06:23:30.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:23:30.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:23:30.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-06-03T06:23:30.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:23:30.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-06-03T06:23:30.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-06-03T06:23:30.969 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672978 2024-06-03T06:23:30.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:23:31.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672978 2024-06-03T06:23:31.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:23:31.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-06-03T06:23:31.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:23:31.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:23:31.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-06-03T06:23:31.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:23:31.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-06-03T06:23:31.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-06-03T06:23:31.304 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509456 2024-06-03T06:23:31.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:23:31.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-06-03T06:23:31.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:23:31.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:23:31.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:23:32.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T06:23:32.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:23:32.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:23:32.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:23:32.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:23:32.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:23:32.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:23:32.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:23:32.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T06:23:32.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:23:32.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:23:32.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:23:32.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T06:23:32.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:23:32.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:23:32.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:23:32.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-03T06:23:32.708 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:23:08.136+0000 7fab238be640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-03T06:23:32.708 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:23:08.136+0000 7fab238be640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-03T06:23:32.708 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:23:08.136+0000 7fab238be640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-06-03T06:23:32.708 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:23:08.136+0000 7fab238be640 20 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-06-03T06:23:32.708 INFO:tasks.workunit.client.0.smithi039.stdout:2024-06-03T06:23:08.136+0000 7fab238be640 19 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-03T06:23:32.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T06:23:32.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:23:32.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:23:32.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:23:32.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:23:32.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:23:32.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:23:32.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:23:32.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:23:32.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:23:32.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:23:32.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:23:32.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:23:32.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:23:32.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:23:32.819 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:23:32.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:23:32.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:23:32.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:23:32.821 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:23:32.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:23:32.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:23:32.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:23:32.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:23:32.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:23:32.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:23:32.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:23:32.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:23:32.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:23:32.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T06:23:32.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T06:23:32.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T06:23:32.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T06:23:32.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:23:32.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:23:32.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:23:32.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:23:32.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:23:32.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:23:32.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:23:32.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:23:32.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:23:32.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:23:32.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:23:32.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:23:32.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:23:32.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:23:32.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:23:32.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:23:32.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:23:32.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:23:32.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:23:32.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:23:32.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:23:32.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:23:32.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:23:32.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:23:32.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:23:32.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:23:32.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:23:32.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:23:32.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T06:23:32.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:23:32.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:23:32.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:23:32.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T06:23:32.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:23:32.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T06:23:32.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:23:32.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T06:23:32.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-06-03T06:23:32.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-06-03T06:23:32.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-03T06:23:32.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-06-03T06:23:32.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-06-03T06:23:32.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-06-03T06:23:32.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T06:23:32.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T06:23:32.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T06:23:32.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T06:23:32.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T06:23:32.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T06:23:33.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T06:23:33.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:23:33.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:23:33.225 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:23:33.225 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:23:33.225 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:23:33.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:23:33.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T06:23:33.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T06:23:33.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T06:23:33.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:23:33.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:23:33.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T06:23:33.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T06:23:33.428 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:23:33.428 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:23:33.428 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:23:33.428 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:23:33.428 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:23:33.428 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:23:33.428 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:23:33.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:23:33.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T06:23:33.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T06:23:33.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:23:33.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:23:33.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T06:23:33.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T06:23:33.516 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:23:33.516 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:23:33.516 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:23:33.516 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:23:33.517 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:23:33.517 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:23:33.517 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:23:33.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:23:33.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T06:23:33.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-06-03T06:23:33.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T06:23:33.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T06:23:33.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T06:23:33.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T06:23:33.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T06:23:33.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T06:23:34.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T06:23:34.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:23:34.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:23:34.436 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:23:34.437 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:23:34.437 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:23:34.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:23:34.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:23:34.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:23:34.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:23:34.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-06-03T06:23:34.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-03T06:23:34.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:23:34.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:23:34.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:23:34.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T06:23:34.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:23:34.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:23:34.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:23:34.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:23:34.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:23:34.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:23:34.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:23:34.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:23:34.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:23:34.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:23:34.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:23:34.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:23:34.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:23:34.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:23:34.478 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:23:34.478 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:23:34.478 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:23:34.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:23:34.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:23:34.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:23:34.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:23:34.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:23:34.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:23:34.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:23:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:23:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:23:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:23:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:23:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:23:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:23:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:23:34.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:23:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aec0f529-caee-4dbe-96c9-da72444bf060 2024-06-03T06:23:34.482 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 aec0f529-caee-4dbe-96c9-da72444bf060 2024-06-03T06:23:34.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 aec0f529-caee-4dbe-96c9-da72444bf060' 2024-06-03T06:23:34.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:23:34.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBmYV1m96IoHRAAcyDOgza1UB9x6u1fwFjTow== 2024-06-03T06:23:34.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBmYV1m96IoHRAAcyDOgza1UB9x6u1fwFjTow=="}' 2024-06-03T06:23:34.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aec0f529-caee-4dbe-96c9-da72444bf060 -i td/osd-scrub-repair/0/new.json 2024-06-03T06:23:34.709 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:23:34.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T06:23:34.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBmYV1m96IoHRAAcyDOgza1UB9x6u1fwFjTow== --osd-uuid aec0f529-caee-4dbe-96c9-da72444bf060 2024-06-03T06:23:34.753 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:34.752+0000 7f490b544640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:23:34.768 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:34.768+0000 7f490b544640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:23:34.769 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:34.769+0000 7f490b544640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:23:34.769 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:34.769+0000 7f490b544640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:23:34.770 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:34.769+0000 7f490b544640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:23:34.770 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:34.769+0000 7f490b544640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:23:34.859 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:34.858+0000 7f490b544640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T06:23:40.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T06:23:40.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:23:40.456 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:23:40.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T06:23:40.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:23:40.843 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:23:40.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T06:23:40.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:23:40.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:23:40.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:23:40.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:23:40.875 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:40.874+0000 7ff872908640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:23:40.877 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:40.876+0000 7ff872908640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:23:40.878 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:40.877+0000 7ff872908640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:23:41.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T06:23:41.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:23:41.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:23:41.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:23:41.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:23:41.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:41.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:23:41.140 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:23:41.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:41.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:41.394 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:41.394+0000 7ff872908640 -1 Falling back to public interface 2024-06-03T06:23:41.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:42.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:42.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:42.432 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:23:42.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:23:42.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:42.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:42.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:43.723 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:23:43.723+0000 7ff872908640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:23:43.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:43.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:43.735 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:23:43.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:23:43.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:43.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:44.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:45.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:45.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:45.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:23:45.159 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:23:45.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:45.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:45.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:46.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:46.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:46.484 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:23:46.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:23:46.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:46.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:46.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:47.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:47.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:47.785 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:23:47.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:23:47.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:47.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:48.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:49.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:49.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:49.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:23:49.091 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:23:49.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:49.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:49.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:50.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:50.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:50.427 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:23:50.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:23:50.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:50.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:50.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:51.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:51.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:51.726 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:23:51.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:23:51.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:51.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:52.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:53.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:53.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:53.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:23:53.027 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:23:53.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:53.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:53.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:54.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:54.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:54.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:23:54.328 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:23:54.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:54.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:54.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:55.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:55.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:55.634 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:23:55.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:23:55.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:55.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:56.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:57.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:57.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:57.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:23:57.168 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:23:57.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:57.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:57.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:23:58.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:23:58.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:23:58.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:23:58.969 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:23:58.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:23:58.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:23:59.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:00.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:00.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:00.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:24:00.516 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:24:00.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:00.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:24:01.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:02.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:02.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:02.261 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:24:02.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:24:02.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:02.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:24:02.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:03.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:03.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:03.738 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:24:03.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:24:03.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:03.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:24:04.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:05.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:05.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:05.486 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:24:05.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:24:05.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:05.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:24:05.808 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3650266038,v1:127.0.0.1:6803/3650266038] [v2:127.0.0.1:6804/3650266038,v1:127.0.0.1:6805/3650266038] exists,up aec0f529-caee-4dbe-96c9-da72444bf060 2024-06-03T06:24:05.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:24:05.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:24:05.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:24:05.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-03T06:24:05.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:24:05.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:24:05.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:24:05.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T06:24:05.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:24:05.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:24:05.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:24:05.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:24:05.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:24:05.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:24:05.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:24:05.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:24:05.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:24:05.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:24:05.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:24:05.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:24:05.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:24:05.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:24:05.814 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:24:05.814 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:24:05.814 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:24:05.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:24:05.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:24:05.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:24:05.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:24:05.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:24:05.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:24:05.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:24:05.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:24:05.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:24:05.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:24:05.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:24:05.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:24:05.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:24:05.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:24:05.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:24:05.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=829dc546-1aa3-4d26-9687-9ba4526e199a 2024-06-03T06:24:05.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 829dc546-1aa3-4d26-9687-9ba4526e199a' 2024-06-03T06:24:05.816 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 829dc546-1aa3-4d26-9687-9ba4526e199a 2024-06-03T06:24:05.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:24:05.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCFYV1m2+aEMRAAGsKQ+uMIomzi0cdUW5RbFA== 2024-06-03T06:24:05.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCFYV1m2+aEMRAAGsKQ+uMIomzi0cdUW5RbFA=="}' 2024-06-03T06:24:05.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 829dc546-1aa3-4d26-9687-9ba4526e199a -i td/osd-scrub-repair/1/new.json 2024-06-03T06:24:06.216 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:24:06.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T06:24:06.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCFYV1m2+aEMRAAGsKQ+uMIomzi0cdUW5RbFA== --osd-uuid 829dc546-1aa3-4d26-9687-9ba4526e199a 2024-06-03T06:24:06.261 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:06.261+0000 7f6e2c01f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:24:06.292 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:06.292+0000 7f6e2c01f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:24:06.293 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:06.293+0000 7f6e2c01f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:24:06.294 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:06.293+0000 7f6e2c01f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:24:06.294 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:06.293+0000 7f6e2c01f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:24:06.294 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:06.293+0000 7f6e2c01f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:24:06.363 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:06.363+0000 7f6e2c01f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T06:24:12.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T06:24:12.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:24:12.169 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:24:12.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T06:24:12.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:24:12.549 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:24:12.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T06:24:12.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:24:12.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:24:12.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:24:12.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:24:12.581 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:12.579+0000 7fe74e017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:24:12.583 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:12.581+0000 7fe74e017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:24:12.584 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:12.582+0000 7fe74e017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:24:12.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T06:24:12.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:24:12.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:24:12.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:24:12.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:24:12.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:12.840 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:24:12.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:24:12.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:12.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:13.111 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:13.110+0000 7fe74e017640 -1 Falling back to public interface 2024-06-03T06:24:13.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:14.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:14.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:14.153 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:24:14.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:24:14.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:14.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:14.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:15.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:15.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:15.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:24:15.460 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:24:15.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:15.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:15.759 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:15.758+0000 7fe74e017640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:24:15.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:16.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:16.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:16.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:24:16.794 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:24:16.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:16.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:17.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:18.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:18.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:18.094 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:24:18.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:24:18.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:18.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:18.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:19.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:19.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:19.424 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:24:19.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:24:19.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:19.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:19.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:20.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:20.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:20.975 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:24:20.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:24:20.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:20.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:21.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:22.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:22.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:22.328 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:24:22.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:24:22.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:22.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:22.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:23.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:23.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:23.634 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:24:23.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:24:23.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:23.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:23.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:24.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:24.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:24.994 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:24:24.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:24:24.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:24.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:25.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:26.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:26.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:26.294 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:24:26.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:24:26.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:26.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:26.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:27.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:27.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:27.851 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:24:27.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:24:27.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:27.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:28.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:29.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:29.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:29.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:24:29.179 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:24:29.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:29.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:29.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:30.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:30.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:30.801 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:24:30.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:24:30.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:30.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:31.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:32.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:32.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:32.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:24:32.478 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:24:32.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:32.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:33.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:34.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:34.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:34.557 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:24:34.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:24:34.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:34.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:35.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:36.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:36.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:36.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:24:36.200 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:24:36.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:36.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:36.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:37.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:37.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:37.721 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:24:37.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:24:37.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:37.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:38.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:39.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:39.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:39.224 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:24:39.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:24:39.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:39.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:24:39.575 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4045949893,v1:127.0.0.1:6811/4045949893] [v2:127.0.0.1:6812/4045949893,v1:127.0.0.1:6813/4045949893] exists,up 829dc546-1aa3-4d26-9687-9ba4526e199a 2024-06-03T06:24:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:24:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:24:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:24:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-03T06:24:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:24:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:24:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:24:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T06:24:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:24:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:24:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:24:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:24:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:24:39.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:24:39.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:24:39.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:24:39.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:24:39.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:24:39.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:24:39.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:24:39.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:24:39.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:24:39.584 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:24:39.584 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:24:39.584 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:24:39.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:24:39.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:24:39.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:24:39.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:24:39.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:24:39.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:24:39.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:24:39.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:24:39.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:24:39.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:24:39.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:24:39.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:24:39.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:24:39.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:24:39.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:24:39.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a8b735bf-21e9-4bf7-8aae-79072f621f50 2024-06-03T06:24:39.586 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 a8b735bf-21e9-4bf7-8aae-79072f621f50 2024-06-03T06:24:39.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a8b735bf-21e9-4bf7-8aae-79072f621f50' 2024-06-03T06:24:39.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:24:39.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnYV1mKGygIxAA+vmMU40Eyz5hE1AawYfcQg== 2024-06-03T06:24:39.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnYV1mKGygIxAA+vmMU40Eyz5hE1AawYfcQg=="}' 2024-06-03T06:24:39.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a8b735bf-21e9-4bf7-8aae-79072f621f50 -i td/osd-scrub-repair/2/new.json 2024-06-03T06:24:39.949 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:24:39.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-03T06:24:39.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCnYV1mKGygIxAA+vmMU40Eyz5hE1AawYfcQg== --osd-uuid a8b735bf-21e9-4bf7-8aae-79072f621f50 2024-06-03T06:24:39.994 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:39.993+0000 7f8710815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:24:40.051 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:40.050+0000 7f8710815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:24:40.052 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:40.051+0000 7f8710815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:24:40.052 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:40.052+0000 7f8710815640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:24:40.052 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:40.052+0000 7f8710815640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:24:40.052 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:40.052+0000 7f8710815640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:24:40.151 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:40.150+0000 7f8710815640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T06:24:46.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-03T06:24:46.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:24:46.900 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T06:24:46.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T06:24:46.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:24:47.276 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:24:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T06:24:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:24:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:24:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:24:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:24:47.309 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:47.307+0000 7fc96100c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:24:47.309 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:47.309+0000 7fc96100c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:24:47.310 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:47.310+0000 7fc96100c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:24:47.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T06:24:47.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:24:47.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:24:47.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:24:47.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:24:47.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:47.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:24:47.567 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:24:47.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:47.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:24:47.831 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:47.831+0000 7fc96100c640 -1 Falling back to public interface 2024-06-03T06:24:47.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:48.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:48.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:48.860 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:24:48.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:24:48.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:48.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:24:49.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:50.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:50.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:50.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:24:50.159 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:24:50.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:50.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:24:50.176 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:24:50.176+0000 7fc96100c640 -1 osd.2 0 log_to_monitors true 2024-06-03T06:24:50.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:51.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:51.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:51.494 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:24:51.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:24:51.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:51.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:24:51.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:52.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:52.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:52.795 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:24:52.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:24:52.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:52.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:24:53.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:54.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:54.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:54.216 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:24:54.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:24:54.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:54.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:24:54.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:55.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:55.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:55.645 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:24:55.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:24:55.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:55.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:24:56.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:57.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:57.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:57.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:24:57.068 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:24:57.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:57.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:24:57.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:58.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:58.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:58.364 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:24:58.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:24:58.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:58.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:24:58.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:24:59.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:24:59.686 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:24:59.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:24:59.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:24:59.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:24:59.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:25:00.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:25:01.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:25:01.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:25:01.135 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:25:01.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:25:01.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:25:01.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:25:01.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:25:02.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:25:02.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:25:02.438 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:25:02.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:25:02.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:25:02.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:25:02.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:25:03.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:25:03.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:25:03.769 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:25:03.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:25:03.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:25:03.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:25:04.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:25:05.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:25:05.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:25:05.541 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:25:05.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:25:05.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:25:05.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:25:05.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:25:06.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:25:06.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:25:06.899 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:25:06.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:25:06.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:25:06.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:25:07.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:25:08.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:25:08.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:25:08.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:25:08.828 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:25:08.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:25:08.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:25:09.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:25:10.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:25:10.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:25:10.247 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:25:10.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:25:10.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:25:10.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:25:11.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:25:12.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:25:12.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:25:12.034 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:25:12.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:25:12.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:25:12.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:25:12.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:25:13.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:25:13.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:25:13.520 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:25:13.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:25:13.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:25:13.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:25:13.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:25:14.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:25:14.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:25:14.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:25:14.832 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:25:14.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:25:14.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:25:15.132 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 34 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3553632885,v1:127.0.0.1:6819/3553632885] [v2:127.0.0.1:6820/3553632885,v1:127.0.0.1:6821/3553632885] exists,up a8b735bf-21e9-4bf7-8aae-79072f621f50 2024-06-03T06:25:15.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:25:15.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:25:15.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:25:15.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-06-03T06:25:15.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T06:25:15.446 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-06-03T06:25:15.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T06:25:15.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T06:25:15.859 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T06:25:15.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:25:16.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T06:25:18.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-06-03T06:25:18.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:25:18.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:25:18.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:25:18.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:25:18.751 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:25:18.751 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:25:18.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:25:18.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:25:18.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:25:18.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:25:18.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:25:18.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:25:18.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:25:18.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:25:18.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:25:19.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:25:19.144 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:25:19.144 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:25:19.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:25:19.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:25:19.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:25:19.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-06-03T06:25:19.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-06-03T06:25:19.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-06-03T06:25:19.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:25:19.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:25:19.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476750 2024-06-03T06:25:19.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476750 2024-06-03T06:25:19.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-68719476750' 2024-06-03T06:25:19.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:25:19.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:25:19.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888067 2024-06-03T06:25:19.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888067 2024-06-03T06:25:19.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-68719476750 2-146028888067' 2024-06-03T06:25:19.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:25:19.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-06-03T06:25:19.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:25:19.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:25:19.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-06-03T06:25:19.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:25:19.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-06-03T06:25:19.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-06-03T06:25:19.512 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836496 2024-06-03T06:25:19.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:25:19.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836496 2024-06-03T06:25:19.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:25:20.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:25:20.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:25:21.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-06-03T06:25:21.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:25:21.126 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476750 2024-06-03T06:25:21.126 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:25:21.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:25:21.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476750 2024-06-03T06:25:21.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:25:21.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476750 2024-06-03T06:25:21.129 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 68719476750 2024-06-03T06:25:21.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476750' 2024-06-03T06:25:21.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:25:21.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476750 -lt 68719476750 2024-06-03T06:25:21.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:25:21.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888067 2024-06-03T06:25:21.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:25:21.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:25:21.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888067 2024-06-03T06:25:21.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:25:21.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888067 2024-06-03T06:25:21.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888067' 2024-06-03T06:25:21.420 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 146028888067 2024-06-03T06:25:21.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:25:21.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888067 2024-06-03T06:25:21.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:25:21.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:25:21.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:25:22.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T06:25:22.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:25:22.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:25:22.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:25:22.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:25:22.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:25:22.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:25:22.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:25:22.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:25:22.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:25:22.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:25:22.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:25:22.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T06:25:22.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:25:22.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:25:22.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-06-03T06:25:22.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-03T06:25:22.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-03T06:25:22.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-06-03T06:25:22.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-03T06:25:22.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-06-03T06:25:23.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-03T06:25:23.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-03T06:25:23.935 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'ecpool' created 2024-06-03T06:25:23.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:25:24.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-06-03T06:25:24.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-03T06:25:24.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:25:24.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:25:24.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:25:24.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:25:24.951 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:25:24.951 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:25:24.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:25:24.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:25:24.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:25:25.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:25:25.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:25:25.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:25:25.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:25:25.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:25:25.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:25:25.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:25:25.490 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:25:25.490 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:25:25.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:25:25.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:25:25.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:25:25.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-06-03T06:25:25.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-06-03T06:25:25.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-06-03T06:25:25.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:25:25.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:25:25.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476752 2024-06-03T06:25:25.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476752 2024-06-03T06:25:25.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-68719476752' 2024-06-03T06:25:25.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:25:25.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:25:25.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888070 2024-06-03T06:25:25.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888070 2024-06-03T06:25:25.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-68719476752 2-146028888070' 2024-06-03T06:25:25.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:25:25.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-06-03T06:25:25.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:25:25.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:25:25.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-06-03T06:25:25.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:25:25.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-06-03T06:25:25.850 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836498 2024-06-03T06:25:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-06-03T06:25:25.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:25:26.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-06-03T06:25:26.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:25:27.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:25:27.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:25:27.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836498 2024-06-03T06:25:27.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:25:27.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476752 2024-06-03T06:25:27.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:25:27.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:25:27.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476752 2024-06-03T06:25:27.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:25:27.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476752 2024-06-03T06:25:27.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476752' 2024-06-03T06:25:27.446 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 68719476752 2024-06-03T06:25:27.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:25:27.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476752 -lt 68719476752 2024-06-03T06:25:27.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:25:27.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888070 2024-06-03T06:25:27.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:25:27.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:25:27.882 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888070 2024-06-03T06:25:27.882 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:25:27.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888070 2024-06-03T06:25:27.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888070' 2024-06-03T06:25:27.884 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 146028888070 2024-06-03T06:25:27.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:25:28.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888070 -lt 146028888070 2024-06-03T06:25:28.174 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:25:28.174 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:25:28.174 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:25:28.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:25:28.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:25:28.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:25:28.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:25:28.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:25:28.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:25:28.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:25:28.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:25:28.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:25:28.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:25:28.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:25:28.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:25:29.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:25:29.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:25:29.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:25:29.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-03T06:25:29.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-06-03T06:25:29.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-06-03T06:25:29.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-03T06:25:29.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-06-03T06:25:29.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-06-03T06:25:29.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:25:29.341 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-06-03T06:25:29.341 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-03T06:25:29.341 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:25:29.341 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:25:29.341 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:25:29.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-06-03T06:25:29.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:25:29.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-06-03T06:25:30.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:25:30.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:25:30.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:25:30.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:25:30.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:25:30.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:25:30.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:25:30.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:25:30.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:25:30.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:25:30.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:25:30.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:25:30.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:25:30.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:25:30.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:25:30.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:25:31.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:25:31.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:25:31.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:25:31.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:25:31.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:25:31.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:25:31.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:25:31.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-03T06:25:32.524 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-03T06:25:32.525 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-06-03T06:25:32.525 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-03T06:25:33.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:25:33.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:25:33.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:25:33.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:25:33.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:25:33.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:25:33.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:25:33.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:25:33.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:25:33.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:25:33.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:25:33.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:25:33.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:25:33.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:25:33.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:25:33.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:25:33.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:25:33.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:25:33.047 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:25:33.047 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:25:33.047 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:25:33.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:25:33.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:25:33.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:25:33.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:25:33.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:25:33.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:25:33.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:25:33.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:25:33.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:25:33.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:25:33.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:25:33.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:25:33.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:25:33.048 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:25:33.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:25:33.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:25:33.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:25:33.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:25:33.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:25:33.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:25:33.081 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:25:33.078+0000 7f562df87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:25:33.081 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:25:33.081+0000 7f562df87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:25:33.083 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:25:33.082+0000 7f562df87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:25:33.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:25:33.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:25:33.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:25:33.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:25:33.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:25:33.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:25:33.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:25:33.345 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:25:33.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:25:33.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:25:33.611 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:25:33.611+0000 7f562df87640 -1 Falling back to public interface 2024-06-03T06:25:33.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:25:34.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:25:34.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:25:34.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:25:34.648 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:25:34.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:25:34.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:25:34.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:25:35.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:25:35.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:25:35.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:25:35.947 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:25:35.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:25:35.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:25:36.198 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:25:36.198+0000 7f562df87640 -1 osd.1 55 log_to_monitors true 2024-06-03T06:25:36.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:25:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:25:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:25:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:25:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:25:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:25:37.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:25:37.710 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 62 up_thru 62 down_at 56 last_clean_interval [16,55) [v2:127.0.0.1:6810/3843184881,v1:127.0.0.1:6811/3843184881] [v2:127.0.0.1:6812/3843184881,v1:127.0.0.1:6813/3843184881] exists,up 829dc546-1aa3-4d26-9687-9ba4526e199a 2024-06-03T06:25:37.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:25:37.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:25:37.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:25:37.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:25:37.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:25:37.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:25:37.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:25:37.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:25:37.711 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:25:37.711 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:25:37.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:25:37.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:25:37.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:25:37.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:25:37.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:25:37.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:25:37.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:25:37.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:25:37.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:25:38.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:25:38.103 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:25:38.103 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:25:38.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:25:38.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:25:38.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:25:38.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-06-03T06:25:38.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-06-03T06:25:38.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-06-03T06:25:38.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:25:38.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:25:38.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972354 2024-06-03T06:25:38.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972354 2024-06-03T06:25:38.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-266287972354' 2024-06-03T06:25:38.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:25:38.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:25:38.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888073 2024-06-03T06:25:38.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888073 2024-06-03T06:25:38.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-266287972354 2-146028888073' 2024-06-03T06:25:38.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:25:38.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-06-03T06:25:38.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:25:38.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:25:38.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-06-03T06:25:38.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:25:38.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-06-03T06:25:38.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-06-03T06:25:38.544 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836502 2024-06-03T06:25:38.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:25:38.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-06-03T06:25:38.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:25:38.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-266287972354 2024-06-03T06:25:38.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:25:38.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:25:38.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-266287972354 2024-06-03T06:25:38.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:25:38.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972354 2024-06-03T06:25:38.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 266287972354' 2024-06-03T06:25:38.857 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 266287972354 2024-06-03T06:25:38.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:25:39.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972354 -lt 266287972354 2024-06-03T06:25:39.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:25:39.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888073 2024-06-03T06:25:39.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:25:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:25:39.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888073 2024-06-03T06:25:39.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:25:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888073 2024-06-03T06:25:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888073' 2024-06-03T06:25:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 146028888073 2024-06-03T06:25:39.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:25:39.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888073 -lt 146028888073 2024-06-03T06:25:39.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:25:39.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:25:39.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:25:39.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:25:39.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:25:39.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:25:39.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:25:39.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:25:39.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:25:39.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:25:39.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:25:40.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:25:40.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:25:40.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:25:40.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:25:40.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:25:40.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:25:40.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:25:40.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-06-03T06:25:40.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:25:40.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:25:40.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:25:40.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:25:40.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-06-03T06:25:40.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-06-03T06:25:40.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:25:40.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:25:40.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:25:40.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:25:41.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-06-03T06:25:39.164326+0000 2024-06-03T06:25:41.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:25:41.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:25:39.164326+0000 2024-06-03T06:25:41.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:25:41.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:25:41.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:25:41.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:25:41.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:25:41.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:25:41.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:25:41.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:25:41.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:25:39.164326+0000 '>' 2024-06-03T06:25:39.164326+0000 2024-06-03T06:25:41.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:25:42.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:25:42.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:25:42.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:25:42.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:25:42.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:25:42.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:25:42.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:25:42.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:25:39.164326+0000 '>' 2024-06-03T06:25:39.164326+0000 2024-06-03T06:25:42.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:25:43.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:25:43.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:25:43.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:25:43.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:25:43.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:25:43.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:25:43.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:25:44.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:25:39.164326+0000 '>' 2024-06-03T06:25:39.164326+0000 2024-06-03T06:25:44.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:25:45.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:25:45.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:25:45.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:25:45.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:25:45.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:25:45.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:25:45.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:25:45.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:25:39.164326+0000 '>' 2024-06-03T06:25:39.164326+0000 2024-06-03T06:25:45.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:25:46.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:25:46.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:25:46.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:25:46.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:25:46.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:25:46.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:25:46.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:25:46.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:25:39.164326+0000 '>' 2024-06-03T06:25:39.164326+0000 2024-06-03T06:25:46.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:25:47.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:25:47.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:25:47.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:25:47.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:25:47.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:25:47.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:25:47.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:25:47.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:25:39.164326+0000 '>' 2024-06-03T06:25:39.164326+0000 2024-06-03T06:25:47.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:25:48.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:25:48.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:25:48.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:25:48.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:25:48.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:25:48.932 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:25:48.932 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:25:49.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:25:39.164326+0000 '>' 2024-06-03T06:25:39.164326+0000 2024-06-03T06:25:49.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:25:50.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:25:50.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:25:50.233 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:25:50.233 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:25:50.233 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:25:50.233 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:25:50.233 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:25:50.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:25:39.164326+0000 '>' 2024-06-03T06:25:39.164326+0000 2024-06-03T06:25:50.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:25:51.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:25:51.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:25:51.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:25:51.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:25:51.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:25:51.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:25:51.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:25:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:25:45.092958+0000 '>' 2024-06-03T06:25:39.164326+0000 2024-06-03T06:25:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:25:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-06-03T06:25:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:25:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:25:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:25:51.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:25:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:25:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:25:51.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:25:51.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:25:51.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:25:51.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:25:51.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:25:51.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:25:51.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:25:51.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:25:51.932 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:25:52.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:25:52.225 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:25:52.226 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:25:52.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:25:52.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:25:52.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:25:52.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-06-03T06:25:52.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-06-03T06:25:52.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-06-03T06:25:52.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:25:52.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:25:52.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972358 2024-06-03T06:25:52.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972358 2024-06-03T06:25:52.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-266287972358' 2024-06-03T06:25:52.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:25:52.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:25:52.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888077 2024-06-03T06:25:52.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888077 2024-06-03T06:25:52.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-266287972358 2-146028888077' 2024-06-03T06:25:52.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:25:52.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:25:52.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-06-03T06:25:52.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:25:52.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-06-03T06:25:52.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:25:52.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-06-03T06:25:52.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-06-03T06:25:52.590 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836506 2024-06-03T06:25:52.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:25:52.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-06-03T06:25:52.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:25:52.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-266287972358 2024-06-03T06:25:52.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:25:52.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:25:52.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-266287972358 2024-06-03T06:25:52.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:25:52.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972358 2024-06-03T06:25:52.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 266287972358' 2024-06-03T06:25:52.881 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 266287972358 2024-06-03T06:25:52.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:25:53.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972358 -lt 266287972358 2024-06-03T06:25:53.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:25:53.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888077 2024-06-03T06:25:53.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:25:53.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:25:53.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888077 2024-06-03T06:25:53.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:25:53.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888077 2024-06-03T06:25:53.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888077' 2024-06-03T06:25:53.173 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 146028888077 2024-06-03T06:25:53.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:25:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888076 -lt 146028888077 2024-06-03T06:25:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:25:54.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:25:54.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:25:54.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888077 -lt 146028888077 2024-06-03T06:25:54.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:25:54.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:25:54.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:25:55.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:25:55.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:25:55.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:25:55.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:25:55.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:25:55.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:25:55.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:25:55.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:25:55.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:25:55.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:25:55.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:25:55.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:25:55.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:25:55.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:25:55.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:25:55.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-06-03T06:25:55.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-06-03T06:25:55.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:25:55.921 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-06-03T06:25:55.921 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-03T06:25:55.921 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:25:55.922 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:25:55.922 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:25:56.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-06-03T06:25:56.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:25:56.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-06-03T06:25:56.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:25:56.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:25:56.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:25:56.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:25:56.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:25:56.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:25:56.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:25:56.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:25:56.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:25:56.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:25:56.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:25:56.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:25:56.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:25:56.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:25:56.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:25:56.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:25:56.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:25:56.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:25:56.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:25:56.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:25:56.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:25:56.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:25:56.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:25:56.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-03T06:25:58.387 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-03T06:25:58.387 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-06-03T06:25:58.387 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:25:58.387 INFO:tasks.workunit.client.0.smithi039.stdout:hinfo_key 2024-06-03T06:25:58.387 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:25:58.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:25:58.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:25:58.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:25:58.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:25:58.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:25:58.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:25:58.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:25:58.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:25:58.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:25:58.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:25:58.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:25:58.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:25:58.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:25:58.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:25:58.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:25:58.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:25:58.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:25:58.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:25:58.666 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:25:58.666 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:25:58.666 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:25:58.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:25:58.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:25:58.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:25:58.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:25:58.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:25:58.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:25:58.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:25:58.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:25:58.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:25:58.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:25:58.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:25:58.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:25:58.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:25:58.668 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:25:58.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:25:58.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:25:58.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:25:58.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:25:58.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:25:58.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:25:58.701 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:25:58.699+0000 7fa4689f8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:25:58.704 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:25:58.702+0000 7fa4689f8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:25:58.705 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:25:58.703+0000 7fa4689f8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:25:58.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:25:58.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:25:58.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:25:58.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:25:58.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:25:58.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:25:58.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:25:58.972 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:25:58.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:25:58.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:25:59.218 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:25:59.217+0000 7fa4689f8640 -1 Falling back to public interface 2024-06-03T06:25:59.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:00.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:00.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:00.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:26:00.320 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:26:00.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:00.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:26:00.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:01.478 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:01.477+0000 7fa4689f8640 -1 osd.1 74 log_to_monitors true 2024-06-03T06:26:01.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:01.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:01.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:26:01.627 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:26:01.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:01.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:26:01.994 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 80 up_thru 62 down_at 75 last_clean_interval [62,74) [v2:127.0.0.1:6810/4023810930,v1:127.0.0.1:6811/4023810930] [v2:127.0.0.1:6812/4023810930,v1:127.0.0.1:6813/4023810930] exists,up 829dc546-1aa3-4d26-9687-9ba4526e199a 2024-06-03T06:26:01.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:26:01.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:26:01.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:26:01.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:26:01.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:26:01.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:26:01.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:26:01.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:26:01.995 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:26:01.995 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:26:01.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:26:01.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:26:01.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:26:02.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:26:02.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:26:02.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:26:02.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:26:02.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:26:02.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:26:02.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:26:02.478 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:26:02.478 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:26:02.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:26:02.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:26:02.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:26:02.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-06-03T06:26:02.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-06-03T06:26:02.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-06-03T06:26:02.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:26:02.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:26:02.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383682 2024-06-03T06:26:02.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383682 2024-06-03T06:26:02.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-343597383682' 2024-06-03T06:26:02.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:26:02.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:26:03.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888080 2024-06-03T06:26:03.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888080 2024-06-03T06:26:03.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-343597383682 2-146028888080' 2024-06-03T06:26:03.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:26:03.126 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-06-03T06:26:03.126 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:26:03.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:26:03.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-06-03T06:26:03.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:26:03.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-06-03T06:26:03.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-06-03T06:26:03.128 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836509 2024-06-03T06:26:03.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:26:03.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-06-03T06:26:03.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:26:04.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:26:04.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:26:04.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-06-03T06:26:04.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:26:04.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-343597383682 2024-06-03T06:26:04.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:26:04.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:26:04.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-343597383682 2024-06-03T06:26:04.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:26:04.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383682 2024-06-03T06:26:04.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 343597383682' 2024-06-03T06:26:04.859 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 343597383682 2024-06-03T06:26:04.859 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:26:05.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383682 -lt 343597383682 2024-06-03T06:26:05.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:26:05.205 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888080 2024-06-03T06:26:05.205 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:26:05.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:26:05.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888080 2024-06-03T06:26:05.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:26:05.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888080 2024-06-03T06:26:05.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888080' 2024-06-03T06:26:05.208 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 146028888080 2024-06-03T06:26:05.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:26:05.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888080 -lt 146028888080 2024-06-03T06:26:05.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:26:05.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:26:05.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:26:05.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:26:05.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:26:05.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:26:05.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:26:05.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:26:05.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:26:05.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:26:05.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:26:06.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-03T06:26:06.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:26:06.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:26:06.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:26:06.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-03T06:26:06.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-03T06:26:06.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:26:06.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-03T06:26:06.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:26:06.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:26:06.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:26:06.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:26:06.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:26:06.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:26:06.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:26:06.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:26:06.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:26:06.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:26:07.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:26:07.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:26:07.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:26:07.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:26:07.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:26:07.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:26:07.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:26:07.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:26:07.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:26:07.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T06:26:07.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:26:07.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:26:07.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:26:07.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:26:07.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:26:07.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:26:07.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:26:07.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:26:07.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:26:07.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:26:07.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:26:07.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:26:07.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:26:07.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:26:07.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:26:07.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:26:07.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:26:07.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:26:07.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:26:07.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:26:07.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:26:07.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:26:07.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:26:07.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:26:07.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:26:07.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:26:07.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:26:07.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:26:07.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T06:26:07.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T06:26:07.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T06:26:07.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T06:26:07.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:26:07.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:26:07.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:26:07.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:26:07.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:26:07.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:26:07.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:26:07.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:26:07.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:26:07.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:26:07.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:26:07.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:26:07.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:26:07.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:26:07.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:26:07.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:26:07.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:26:07.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:26:07.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:26:07.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:26:07.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:26:07.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:26:07.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:26:07.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:26:07.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:26:07.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:26:07.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:26:07.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:26:07.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T06:26:07.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:26:07.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:26:07.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:26:07.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T06:26:07.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:26:07.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T06:26:07.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:26:07.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T06:26:07.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-06-03T06:26:07.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-06-03T06:26:07.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-06-03T06:26:07.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-03T06:26:07.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-06-03T06:26:07.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-06-03T06:26:07.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-06-03T06:26:07.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T06:26:07.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T06:26:07.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T06:26:07.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T06:26:07.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T06:26:07.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T06:26:07.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T06:26:07.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:26:07.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:26:07.947 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:26:07.947 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:26:07.947 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:26:07.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:26:07.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T06:26:08.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T06:26:08.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T06:26:08.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:26:08.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:26:08.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T06:26:08.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T06:26:08.141 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:26:08.141 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:26:08.141 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:26:08.141 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:26:08.141 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:26:08.142 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:26:08.142 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:26:08.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:26:08.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T06:26:08.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T06:26:08.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:26:08.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:26:08.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T06:26:08.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T06:26:08.226 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:26:08.226 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:26:08.226 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:26:08.227 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:26:08.227 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:26:08.227 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:26:08.227 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:26:08.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:26:08.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T06:26:08.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-06-03T06:26:08.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T06:26:08.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T06:26:08.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T06:26:08.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T06:26:08.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T06:26:08.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T06:26:09.133 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T06:26:09.133 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:26:09.133 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:26:09.133 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:26:09.133 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:26:09.133 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:26:09.133 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:26:09.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:26:09.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:26:09.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:26:09.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-06-03T06:26:09.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-03T06:26:09.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:26:09.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:26:09.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:26:09.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T06:26:09.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:26:09.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:26:09.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:26:09.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:26:09.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:26:09.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:26:09.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:26:09.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:26:09.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:26:09.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:26:09.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:26:09.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:26:09.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:26:09.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:26:09.168 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:26:09.168 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:26:09.168 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:26:09.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:26:09.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:26:09.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:26:09.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:26:09.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:26:09.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:26:09.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:26:09.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:26:09.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:26:09.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:26:09.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:26:09.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:26:09.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:26:09.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:26:09.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:26:09.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3c743e11-467b-4813-b90e-1191d2dec3ed 2024-06-03T06:26:09.170 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 3c743e11-467b-4813-b90e-1191d2dec3ed 2024-06-03T06:26:09.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3c743e11-467b-4813-b90e-1191d2dec3ed' 2024-06-03T06:26:09.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:26:09.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQABYl1mDkYgCxAAFeSHvfukH8j0/uvJxRx9FA== 2024-06-03T06:26:09.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQABYl1mDkYgCxAAFeSHvfukH8j0/uvJxRx9FA=="}' 2024-06-03T06:26:09.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3c743e11-467b-4813-b90e-1191d2dec3ed -i td/osd-scrub-repair/0/new.json 2024-06-03T06:26:09.439 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:26:09.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T06:26:09.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQABYl1mDkYgCxAAFeSHvfukH8j0/uvJxRx9FA== --osd-uuid 3c743e11-467b-4813-b90e-1191d2dec3ed 2024-06-03T06:26:09.483 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:09.482+0000 7fd6569b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:26:09.507 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:09.506+0000 7fd6569b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:26:09.508 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:09.508+0000 7fd6569b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:26:09.509 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:09.508+0000 7fd6569b8640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:26:09.509 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:09.508+0000 7fd6569b8640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:26:09.509 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:09.508+0000 7fd6569b8640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:26:09.599 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:09.598+0000 7fd6569b8640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T06:26:15.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T06:26:15.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:26:15.090 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:26:15.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T06:26:15.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:26:15.471 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:26:15.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T06:26:15.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:26:15.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:26:15.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:26:15.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:26:15.503 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:15.503+0000 7f6cff80e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:26:15.508 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:15.508+0000 7f6cff80e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:26:15.509 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:15.509+0000 7f6cff80e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:26:15.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T06:26:15.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:26:15.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:26:15.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:26:15.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:26:15.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:15.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:26:15.763 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:26:15.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:15.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:16.027 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:16.027+0000 7f6cff80e640 -1 Falling back to public interface 2024-06-03T06:26:16.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:17.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:17.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:17.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:26:17.095 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:26:17.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:17.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:17.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:18.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:18.399 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:26:18.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:18.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:26:18.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:18.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:18.400 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:18.400+0000 7f6cff80e640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:26:18.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:19.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:19.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:19.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:26:19.828 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:26:19.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:19.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:20.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:21.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:21.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:21.167 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:26:21.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:26:21.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:21.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:21.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:22.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:22.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:22.468 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:26:22.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:26:22.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:22.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:22.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:23.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:23.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:23.771 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:26:23.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:26:23.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:23.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:24.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:25.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:25.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:25.169 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:26:25.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:26:25.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:25.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:25.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:26.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:26.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:26.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:26:26.470 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:26:26.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:26.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:26.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:27.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:27.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:27.771 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:26:27.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:26:27.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:27.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:28.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:29.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:29.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:29.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:26:29.135 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:26:29.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:29.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:29.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:30.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:30.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:30.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:26:30.438 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:26:30.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:30.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:31.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:32.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:32.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:32.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:26:32.015 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:26:32.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:32.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:32.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:33.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:33.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:33.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:26:33.563 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:26:33.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:33.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:33.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:34.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:34.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:34.969 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:26:34.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:26:34.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:34.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:35.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:36.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:36.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:36.393 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:26:36.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:26:36.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:36.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:36.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:26:37.867 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:26:37.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:37.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:38.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:39.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:39.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:39.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:26:39.526 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:26:39.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:39.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:39.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:40.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:40.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:40.969 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:26:40.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:26:40.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:40.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:26:41.270 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1281313359,v1:127.0.0.1:6803/1281313359] [v2:127.0.0.1:6804/1281313359,v1:127.0.0.1:6805/1281313359] exists,up 3c743e11-467b-4813-b90e-1191d2dec3ed 2024-06-03T06:26:41.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:26:41.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:26:41.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:26:41.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-03T06:26:41.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:26:41.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:26:41.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:26:41.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T06:26:41.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:26:41.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:26:41.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:26:41.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:26:41.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:26:41.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:26:41.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:26:41.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:26:41.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:26:41.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:26:41.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:26:41.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:26:41.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:26:41.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:26:41.277 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:26:41.277 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:26:41.277 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:26:41.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:26:41.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:26:41.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:26:41.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:26:41.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:26:41.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:26:41.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:26:41.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:26:41.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:26:41.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:26:41.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:26:41.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:26:41.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:26:41.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:26:41.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:26:41.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a5eeb21d-945c-457b-a667-bccc293877b3 2024-06-03T06:26:41.278 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 a5eeb21d-945c-457b-a667-bccc293877b3 2024-06-03T06:26:41.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a5eeb21d-945c-457b-a667-bccc293877b3' 2024-06-03T06:26:41.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:26:41.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhYl1ml8R4ERAAdQMK3msu9/DuTX1lWc6pXw== 2024-06-03T06:26:41.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhYl1ml8R4ERAAdQMK3msu9/DuTX1lWc6pXw=="}' 2024-06-03T06:26:41.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a5eeb21d-945c-457b-a667-bccc293877b3 -i td/osd-scrub-repair/1/new.json 2024-06-03T06:26:41.702 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:26:41.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T06:26:41.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAhYl1ml8R4ERAAdQMK3msu9/DuTX1lWc6pXw== --osd-uuid a5eeb21d-945c-457b-a667-bccc293877b3 2024-06-03T06:26:41.748 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:41.747+0000 7f379c812640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:26:41.870 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:41.870+0000 7f379c812640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:26:41.872 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:41.871+0000 7f379c812640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:26:41.872 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:41.871+0000 7f379c812640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:26:41.873 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:41.871+0000 7f379c812640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:26:41.873 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:41.871+0000 7f379c812640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:26:41.946 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:41.945+0000 7f379c812640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T06:26:47.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T06:26:47.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:26:47.750 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:26:47.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T06:26:47.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:26:48.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T06:26:48.127 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:26:48.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:26:48.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:26:48.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:26:48.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:26:48.157 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:48.156+0000 7f6f3a11c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:26:48.159 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:48.158+0000 7f6f3a11c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:26:48.160 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:48.159+0000 7f6f3a11c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:26:48.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T06:26:48.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:26:48.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:26:48.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:26:48.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:26:48.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:48.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:26:48.422 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:26:48.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:48.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:26:48.674 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:48.673+0000 7f6f3a11c640 -1 Falling back to public interface 2024-06-03T06:26:48.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:49.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:49.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:49.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:26:49.725 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:26:49.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:49.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:26:50.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:51.013 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:26:51.011+0000 7f6f3a11c640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:26:51.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:51.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:51.024 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:26:51.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:26:51.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:51.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:26:51.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:52.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:52.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:52.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:26:52.339 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:26:52.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:52.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:26:52.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:53.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:53.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:53.652 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:26:53.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:26:53.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:53.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:26:54.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:55.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:55.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:55.025 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:26:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:26:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:26:55.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:56.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:56.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:56.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:26:56.326 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:26:56.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:56.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:26:56.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:57.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:57.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:57.681 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:26:57.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:26:57.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:57.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:26:58.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:26:59.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:26:59.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:26:59.117 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:26:59.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:26:59.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:26:59.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:26:59.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:00.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:00.426 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:27:00.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:00.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:27:00.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:00.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:27:00.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:01.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:01.787 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:27:01.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:01.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:27:01.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:01.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:27:02.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:03.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:03.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:03.091 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:27:03.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:27:03.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:03.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:27:03.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:04.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:04.545 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:27:04.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:04.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:27:04.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:04.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:27:05.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:06.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:06.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:06.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:27:06.406 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:27:06.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:06.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:27:07.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:08.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:08.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:08.143 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:27:08.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:27:08.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:08.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:27:08.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:09.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:09.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:09.891 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:27:09.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:27:09.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:09.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:27:10.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:11.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:11.857 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:27:11.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:11.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:27:11.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:11.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:27:12.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:13.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:13.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:13.309 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:27:13.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:27:13.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:13.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:27:13.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:14.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:14.812 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:27:14.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:14.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:27:14.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:14.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:27:15.150 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2104787036,v1:127.0.0.1:6811/2104787036] [v2:127.0.0.1:6812/2104787036,v1:127.0.0.1:6813/2104787036] exists,up a5eeb21d-945c-457b-a667-bccc293877b3 2024-06-03T06:27:15.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:27:15.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:27:15.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:27:15.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-03T06:27:15.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:27:15.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:27:15.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:27:15.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T06:27:15.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:27:15.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:27:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:27:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:27:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:27:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:27:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:27:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:27:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:27:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:27:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:27:15.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:27:15.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:27:15.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:27:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:27:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:27:15.159 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:27:15.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:27:15.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:27:15.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:27:15.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:27:15.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:27:15.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:27:15.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:27:15.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:27:15.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:27:15.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:27:15.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:27:15.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:27:15.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:27:15.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:27:15.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:27:15.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0a5a8720-4118-4819-bb51-34f3a3335b18 2024-06-03T06:27:15.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0a5a8720-4118-4819-bb51-34f3a3335b18' 2024-06-03T06:27:15.162 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 0a5a8720-4118-4819-bb51-34f3a3335b18 2024-06-03T06:27:15.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:27:15.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBDYl1mRrliChAAQ0RB06YdwQoZst8cGGXIBQ== 2024-06-03T06:27:15.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBDYl1mRrliChAAQ0RB06YdwQoZst8cGGXIBQ=="}' 2024-06-03T06:27:15.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0a5a8720-4118-4819-bb51-34f3a3335b18 -i td/osd-scrub-repair/2/new.json 2024-06-03T06:27:15.525 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:27:15.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-03T06:27:15.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBDYl1mRrliChAAQ0RB06YdwQoZst8cGGXIBQ== --osd-uuid 0a5a8720-4118-4819-bb51-34f3a3335b18 2024-06-03T06:27:15.570 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:27:15.570+0000 7f319b1a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:27:15.601 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:27:15.601+0000 7f319b1a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:27:15.602 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:27:15.602+0000 7f319b1a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:27:15.603 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:27:15.602+0000 7f319b1a5640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:27:15.603 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:27:15.602+0000 7f319b1a5640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:27:15.603 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:27:15.602+0000 7f319b1a5640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:27:15.685 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:27:15.684+0000 7f319b1a5640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T06:27:21.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-03T06:27:21.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:27:21.513 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T06:27:21.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T06:27:21.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:27:21.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T06:27:21.888 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:27:21.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:27:21.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:27:21.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:27:21.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:27:21.919 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:27:21.919+0000 7f0350214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:27:21.921 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:27:21.921+0000 7f0350214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:27:21.923 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:27:21.922+0000 7f0350214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:27:22.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T06:27:22.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:27:22.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:27:22.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:27:22.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:27:22.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:22.179 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:27:22.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:27:22.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:22.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:22.441 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:27:22.441+0000 7f0350214640 -1 Falling back to public interface 2024-06-03T06:27:22.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:23.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:23.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:23.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:27:23.470 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:27:23.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:23.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:23.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:24.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:24.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:24.769 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:27:24.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:27:24.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:24.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:24.796 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:27:24.796+0000 7f0350214640 -1 osd.2 0 log_to_monitors true 2024-06-03T06:27:25.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:26.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:26.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:26.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:27:26.107 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:27:26.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:26.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:26.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:27.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:27.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:27.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:27:27.409 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:27:27.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:27.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:27.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:28.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:28.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:28.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:27:28.820 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:27:28.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:28.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:29.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:30.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:30.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:27:30.126 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:27:30.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:30.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:30.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:31.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:31.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:31.475 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:27:31.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:27:31.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:31.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:31.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:32.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:32.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:32.778 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:27:32.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:27:32.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:32.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:33.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:27:34.084 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:27:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:34.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:35.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:35.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:27:35.485 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:27:35.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:35.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:35.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:36.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:36.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:36.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:27:36.789 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:27:36.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:36.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:37.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:38.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:38.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:38.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:27:38.193 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:27:38.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:38.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:38.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:39.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:39.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:39.657 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:27:39.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:27:39.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:39.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:40.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:41.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:41.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:41.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:27:41.462 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:27:41.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:41.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:41.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:42.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:42.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:42.831 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:27:42.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:27:42.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:42.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:43.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:44.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:44.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:44.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:27:44.503 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:27:44.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:44.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:44.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:45.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:45.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:45.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:27:45.996 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:27:45.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:45.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:46.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:47.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:47.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:47.485 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:27:47.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:27:47.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:47.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:48.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:27:49.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:27:49.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:27:49.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:27:49.385 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:27:49.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:27:49.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:27:49.682 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2241085742,v1:127.0.0.1:6819/2241085742] [v2:127.0.0.1:6820/2241085742,v1:127.0.0.1:6821/2241085742] exists,up 0a5a8720-4118-4819-bb51-34f3a3335b18 2024-06-03T06:27:49.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:27:49.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:27:49.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:27:49.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-06-03T06:27:49.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T06:27:49.960 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-06-03T06:27:49.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T06:27:49.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T06:27:50.401 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T06:27:50.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:27:51.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T06:27:53.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-06-03T06:27:53.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:27:53.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:27:53.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:27:53.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:27:53.767 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:27:53.767 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:27:53.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:27:53.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:27:53.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:27:53.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:27:53.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:27:53.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:27:53.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:27:53.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:27:53.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:27:54.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:27:54.225 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:27:54.225 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:27:54.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:27:54.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:27:54.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:27:54.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-06-03T06:27:54.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-06-03T06:27:54.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-06-03T06:27:54.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:27:54.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:27:54.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-06-03T06:27:54.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-06-03T06:27:54.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-64424509453' 2024-06-03T06:27:54.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:27:54.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:27:54.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953478 2024-06-03T06:27:54.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953478 2024-06-03T06:27:54.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-64424509453 2-137438953478' 2024-06-03T06:27:54.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:27:54.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-06-03T06:27:54.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:27:54.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:27:54.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-06-03T06:27:54.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:27:54.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-06-03T06:27:54.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-06-03T06:27:54.589 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836498 2024-06-03T06:27:54.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:27:54.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-06-03T06:27:54.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:27:55.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:27:55.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:27:56.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-06-03T06:27:56.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:27:56.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-64424509453 2024-06-03T06:27:56.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:27:56.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:27:56.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-64424509453 2024-06-03T06:27:56.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:27:56.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-06-03T06:27:56.187 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 64424509453 2024-06-03T06:27:56.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 64424509453' 2024-06-03T06:27:56.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:27:56.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-06-03T06:27:56.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:27:56.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953478 2024-06-03T06:27:56.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:27:56.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:27:56.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953478 2024-06-03T06:27:56.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:27:56.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953478 2024-06-03T06:27:56.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953478' 2024-06-03T06:27:56.497 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 137438953478 2024-06-03T06:27:56.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:27:56.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953478 -lt 137438953478 2024-06-03T06:27:56.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:27:56.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:27:56.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:27:57.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T06:27:57.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:27:57.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:27:57.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:27:57.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:27:57.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:27:57.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:27:57.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:27:57.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:27:57.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:27:57.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:27:57.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:27:57.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T06:27:57.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:27:57.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:27:57.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-06-03T06:27:57.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-03T06:27:57.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-03T06:27:57.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-06-03T06:27:57.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-03T06:27:57.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-06-03T06:27:58.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-03T06:27:58.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-03T06:27:59.480 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'ecpool' created 2024-06-03T06:27:59.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:28:00.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-06-03T06:28:00.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-06-03T06:28:00.997 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 2 allow_ec_overwrites to true 2024-06-03T06:28:01.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-03T06:28:01.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:28:01.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:28:01.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:28:01.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:28:01.010 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:28:01.010 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:28:01.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:28:01.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:28:01.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:28:01.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:28:01.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:28:01.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:28:01.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:28:01.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:28:01.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:28:01.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:28:01.447 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:28:01.448 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:28:01.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:28:01.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:28:01.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:28:01.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-06-03T06:28:01.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-06-03T06:28:01.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-06-03T06:28:01.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:28:01.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:28:01.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-06-03T06:28:01.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-06-03T06:28:01.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-64424509456' 2024-06-03T06:28:01.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:28:01.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:28:01.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953480 2024-06-03T06:28:01.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953480 2024-06-03T06:28:01.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-64424509456 2-137438953480' 2024-06-03T06:28:01.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:28:01.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-06-03T06:28:01.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:28:01.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:28:01.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-06-03T06:28:01.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:28:01.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-06-03T06:28:01.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-06-03T06:28:01.807 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836501 2024-06-03T06:28:01.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:28:02.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-06-03T06:28:02.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:28:03.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:28:03.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:28:03.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-06-03T06:28:03.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:28:03.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-64424509456 2024-06-03T06:28:03.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:28:03.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:28:03.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-64424509456 2024-06-03T06:28:03.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:28:03.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-06-03T06:28:03.397 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 64424509456 2024-06-03T06:28:03.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 64424509456' 2024-06-03T06:28:03.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:28:03.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-06-03T06:28:03.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:28:03.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953480 2024-06-03T06:28:03.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:28:03.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:28:03.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953480 2024-06-03T06:28:03.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:28:03.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953480 2024-06-03T06:28:03.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953480' 2024-06-03T06:28:03.706 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 137438953480 2024-06-03T06:28:03.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:28:04.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953480 -lt 137438953480 2024-06-03T06:28:04.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:28:04.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:28:04.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:28:04.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:28:04.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:28:04.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:28:04.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:28:04.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:28:04.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:28:04.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:28:04.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:28:04.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:28:04.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:28:04.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:28:04.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:28:05.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:28:05.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:28:05.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:28:05.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-03T06:28:05.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-06-03T06:28:05.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-06-03T06:28:05.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-03T06:28:05.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-06-03T06:28:05.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-06-03T06:28:05.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:28:05.685 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-06-03T06:28:05.685 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-03T06:28:05.685 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:28:05.685 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:28:05.686 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:28:05.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-06-03T06:28:05.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:28:05.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-06-03T06:28:06.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:28:06.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:28:06.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:28:06.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:28:06.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:28:06.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:28:06.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:28:06.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:28:06.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:28:06.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:28:06.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:28:06.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:28:06.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:28:06.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:28:06.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:28:06.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:28:06.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:28:06.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:28:06.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:28:06.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:28:06.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:28:06.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:28:06.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:28:06.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-03T06:28:08.388 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-03T06:28:08.388 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-06-03T06:28:08.388 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-03T06:28:08.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:28:08.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:28:08.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:28:08.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:28:08.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:28:08.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:28:08.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:28:08.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:28:08.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:28:08.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:28:08.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:28:08.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:28:08.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:28:08.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:28:08.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:28:08.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:28:08.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:28:08.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:28:08.910 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:28:08.910 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:28:08.910 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:28:08.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:28:08.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:28:08.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:28:08.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:28:08.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:28:08.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:28:08.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:28:08.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:28:08.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:28:08.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:28:08.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:28:08.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:28:08.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:28:08.912 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:28:08.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:28:08.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:28:08.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:28:08.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:28:08.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:28:08.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:28:08.944 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:08.944+0000 7f7bb6161640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:28:08.946 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:08.946+0000 7f7bb6161640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:28:08.948 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:08.947+0000 7f7bb6161640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:28:09.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:28:09.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:28:09.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:28:09.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:28:09.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:28:09.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:09.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:28:09.216 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:28:09.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:09.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:28:09.465 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:09.465+0000 7f7bb6161640 -1 Falling back to public interface 2024-06-03T06:28:09.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:10.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:10.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:10.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:28:10.544 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:28:10.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:10.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:28:10.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:11.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:11.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:11.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:28:11.865 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:28:11.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:11.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:28:11.990 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:11.990+0000 7f7bb6161640 -1 osd.1 56 log_to_monitors true 2024-06-03T06:28:12.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:13.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:13.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:13.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:28:13.188 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:28:13.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:13.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:28:13.527 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 63 up_thru 63 down_at 57 last_clean_interval [15,56) [v2:127.0.0.1:6810/3737373111,v1:127.0.0.1:6811/3737373111] [v2:127.0.0.1:6812/3737373111,v1:127.0.0.1:6813/3737373111] exists,up a5eeb21d-945c-457b-a667-bccc293877b3 2024-06-03T06:28:13.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:28:13.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:28:13.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:28:13.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:28:13.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:28:13.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:28:13.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:28:13.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:28:13.529 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:28:13.529 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:28:13.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:28:13.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:28:13.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:28:13.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:28:13.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:28:13.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:28:13.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:28:13.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:28:13.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:28:14.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:28:14.039 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:28:14.039 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:28:14.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:28:14.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:28:14.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:28:14.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-06-03T06:28:14.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-06-03T06:28:14.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-06-03T06:28:14.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:28:14.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:28:14.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=270582939650 2024-06-03T06:28:14.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 270582939650 2024-06-03T06:28:14.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-270582939650' 2024-06-03T06:28:14.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:28:14.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:28:14.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953484 2024-06-03T06:28:14.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953484 2024-06-03T06:28:14.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-270582939650 2-137438953484' 2024-06-03T06:28:14.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:28:14.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-06-03T06:28:14.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:28:14.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:28:14.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-06-03T06:28:14.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:28:14.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-06-03T06:28:14.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-06-03T06:28:14.375 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836504 2024-06-03T06:28:14.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:28:14.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-06-03T06:28:14.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:28:15.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:28:15.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:28:15.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-06-03T06:28:15.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:28:15.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-270582939650 2024-06-03T06:28:15.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:28:15.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:28:15.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-270582939650 2024-06-03T06:28:15.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:28:15.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=270582939650 2024-06-03T06:28:15.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 270582939650' 2024-06-03T06:28:15.992 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 270582939650 2024-06-03T06:28:15.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:28:16.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939650 -lt 270582939650 2024-06-03T06:28:16.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:28:16.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953484 2024-06-03T06:28:16.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:28:16.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:28:16.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953484 2024-06-03T06:28:16.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:28:16.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953484 2024-06-03T06:28:16.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953484' 2024-06-03T06:28:16.290 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 137438953484 2024-06-03T06:28:16.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:28:16.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953484 -lt 137438953484 2024-06-03T06:28:16.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:28:16.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:28:16.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:28:16.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:28:16.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:28:16.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:28:16.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:28:16.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:28:16.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:28:16.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:28:16.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:28:17.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:28:17.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:28:17.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:28:17.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:28:17.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:28:17.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:28:17.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:28:17.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-06-03T06:28:17.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:28:17.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:28:17.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:28:17.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:28:17.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-06-03T06:28:17.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-06-03T06:28:17.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:28:17.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:28:17.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:28:17.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:28:18.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-06-03T06:27:59.346749+0000 2024-06-03T06:28:18.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:28:18.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:27:59.346749+0000 2024-06-03T06:28:18.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:28:18.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:28:18.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:28:18.254 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:28:18.254 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:28:18.254 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:28:18.254 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:28:18.254 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:28:18.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:27:59.346749+0000 '>' 2024-06-03T06:27:59.346749+0000 2024-06-03T06:28:18.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:28:19.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:28:19.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:28:19.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:28:19.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:28:19.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:28:19.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:28:19.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:28:19.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:27:59.346749+0000 '>' 2024-06-03T06:27:59.346749+0000 2024-06-03T06:28:19.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:28:20.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:28:20.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:28:20.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:28:20.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:28:20.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:28:20.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:28:20.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:28:21.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:28:18.832039+0000 '>' 2024-06-03T06:27:59.346749+0000 2024-06-03T06:28:21.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:28:21.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-06-03T06:28:21.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:28:21.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:28:21.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:28:21.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:28:21.228 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:28:21.228 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:28:21.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:28:21.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:28:21.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:28:21.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:28:21.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:28:21.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:28:21.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:28:21.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:28:21.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:28:21.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:28:21.617 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:28:21.617 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:28:21.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:28:21.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:28:21.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:28:21.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-06-03T06:28:21.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-06-03T06:28:21.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-06-03T06:28:21.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:28:21.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:28:21.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=270582939652 2024-06-03T06:28:21.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 270582939652 2024-06-03T06:28:21.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-270582939652' 2024-06-03T06:28:21.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:28:21.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:28:22.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953486 2024-06-03T06:28:22.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953486 2024-06-03T06:28:22.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-270582939652 2-137438953486' 2024-06-03T06:28:22.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:28:22.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-06-03T06:28:22.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:28:22.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:28:22.126 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-06-03T06:28:22.126 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:28:22.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-06-03T06:28:22.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-06-03T06:28:22.128 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836507 2024-06-03T06:28:22.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:28:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-06-03T06:28:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:28:23.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:28:23.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:28:23.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-06-03T06:28:23.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:28:23.720 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-270582939652 2024-06-03T06:28:23.720 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:28:23.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:28:23.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-270582939652 2024-06-03T06:28:23.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:28:23.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=270582939652 2024-06-03T06:28:23.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 270582939652' 2024-06-03T06:28:23.723 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 270582939652 2024-06-03T06:28:23.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:28:24.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939653 -lt 270582939652 2024-06-03T06:28:24.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:28:24.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953486 2024-06-03T06:28:24.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:28:24.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:28:24.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953486 2024-06-03T06:28:24.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:28:24.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953486 2024-06-03T06:28:24.059 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 137438953486 2024-06-03T06:28:24.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953486' 2024-06-03T06:28:24.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:28:24.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953486 -lt 137438953486 2024-06-03T06:28:24.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:28:24.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:28:24.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:28:24.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:28:24.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:28:24.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:28:24.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:28:24.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:28:24.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:28:24.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:28:24.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:28:25.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:28:25.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:28:25.019 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:28:25.019 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:28:25.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:28:25.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:28:25.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:28:25.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-06-03T06:28:25.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-06-03T06:28:25.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:28:25.420 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-06-03T06:28:25.420 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-03T06:28:25.420 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:28:25.420 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:28:25.420 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:28:25.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-06-03T06:28:25.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:28:25.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-06-03T06:28:26.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:28:26.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:28:26.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:28:26.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:28:26.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:28:26.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:28:26.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:28:26.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:28:26.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:28:26.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:28:26.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:28:26.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:28:26.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:28:26.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:28:26.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:28:26.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:28:27.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:28:27.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:28:27.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:28:27.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:28:27.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:28:27.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:28:27.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:28:27.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-03T06:28:28.511 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-03T06:28:28.511 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:28:28.511 INFO:tasks.workunit.client.0.smithi039.stdout:hinfo_key 2024-06-03T06:28:28.512 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:28:28.512 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-06-03T06:28:29.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:28:29.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:28:29.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:28:29.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:28:29.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:28:29.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:28:29.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:28:29.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:28:29.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:28:29.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:28:29.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:28:29.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:28:29.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:28:29.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:28:29.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:28:29.035 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:28:29.035 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:28:29.035 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:28:29.035 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:28:29.035 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:28:29.035 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:28:29.035 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:28:29.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:28:29.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:28:29.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:28:29.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:28:29.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:28:29.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:28:29.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:28:29.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:28:29.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:28:29.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T06:28:29.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:28:29.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:28:29.036 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:28:29.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-03T06:28:29.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:28:29.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:28:29.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:28:29.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:28:29.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:28:29.065 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:29.064+0000 7fc0fc863640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:28:29.067 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:29.066+0000 7fc0fc863640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:28:29.069 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:29.068+0000 7fc0fc863640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:28:29.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:28:29.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:28:29.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:28:29.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:28:29.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:28:29.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:29.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:28:29.334 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:28:29.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:29.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:28:29.590 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:29.589+0000 7fc0fc863640 -1 Falling back to public interface 2024-06-03T06:28:29.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:30.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:30.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:30.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:28:30.621 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:28:30.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:30.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:28:30.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:31.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:31.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:31.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:28:31.921 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:28:31.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:31.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:28:32.255 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:32.253+0000 7fc0fc863640 -1 osd.1 73 log_to_monitors true 2024-06-03T06:28:32.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:33.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:33.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:33.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:28:33.258 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:28:33.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:33.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:28:33.559 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 80 up_thru 80 down_at 74 last_clean_interval [63,73) [v2:127.0.0.1:6810/4048586180,v1:127.0.0.1:6811/4048586180] [v2:127.0.0.1:6812/4048586180,v1:127.0.0.1:6813/4048586180] exists,up a5eeb21d-945c-457b-a667-bccc293877b3 2024-06-03T06:28:33.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:28:33.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:28:33.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:28:33.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:28:33.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:28:33.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:28:33.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:28:33.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:28:33.560 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:28:33.561 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:28:33.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:28:33.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:28:33.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:28:33.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:28:33.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:28:33.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:28:33.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:28:33.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:28:33.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:28:34.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:28:34.026 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:28:34.026 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:28:34.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:28:34.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:28:34.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:28:34.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-06-03T06:28:34.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-06-03T06:28:34.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-06-03T06:28:34.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:28:34.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:28:34.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383682 2024-06-03T06:28:34.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383682 2024-06-03T06:28:34.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-343597383682' 2024-06-03T06:28:34.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:28:34.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:28:34.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953490 2024-06-03T06:28:34.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953490 2024-06-03T06:28:34.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-343597383682 2-137438953490' 2024-06-03T06:28:34.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:28:34.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-06-03T06:28:34.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:28:34.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:28:34.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-06-03T06:28:34.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:28:34.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-06-03T06:28:34.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-06-03T06:28:34.451 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836510 2024-06-03T06:28:34.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:28:34.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836510 2024-06-03T06:28:34.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:28:35.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:28:35.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:28:36.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836510 2024-06-03T06:28:36.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:28:36.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-343597383682 2024-06-03T06:28:36.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:28:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:28:36.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-343597383682 2024-06-03T06:28:36.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:28:36.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383682 2024-06-03T06:28:36.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 343597383682' 2024-06-03T06:28:36.045 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 343597383682 2024-06-03T06:28:36.045 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:28:36.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383682 -lt 343597383682 2024-06-03T06:28:36.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:28:36.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953490 2024-06-03T06:28:36.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:28:36.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:28:36.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953490 2024-06-03T06:28:36.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:28:36.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953490 2024-06-03T06:28:36.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953490' 2024-06-03T06:28:36.349 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 137438953490 2024-06-03T06:28:36.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:28:36.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953490 -lt 137438953490 2024-06-03T06:28:36.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:28:36.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:28:36.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:28:37.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:28:37.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:28:37.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:28:37.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:28:37.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:28:37.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:28:37.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:28:37.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:28:37.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:28:37.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:28:37.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:28:37.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:28:37.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:28:37.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:28:37.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:28:37.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:28:37.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:28:37.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T06:28:37.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:28:37.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:28:37.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:28:37.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:28:37.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:28:37.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:28:37.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:28:37.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:28:37.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:28:37.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:28:37.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:28:37.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:28:37.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:28:37.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:28:37.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:28:37.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:28:37.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:28:37.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:28:37.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:28:37.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:28:37.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:28:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:28:37.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:28:37.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:28:37.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:28:37.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:28:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:28:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:28:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T06:28:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T06:28:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T06:28:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T06:28:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:28:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:28:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:28:37.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:28:37.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:28:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:28:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:28:37.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:28:37.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:28:37.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:28:37.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:28:37.888 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:28:37.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:28:37.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:28:37.889 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:28:37.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:28:37.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:28:37.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:28:37.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:28:37.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:28:37.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:28:37.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:28:37.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:28:37.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:28:37.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:28:37.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:28:37.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:28:37.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:28:37.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T06:28:37.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:28:37.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:28:37.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:28:37.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T06:28:37.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:28:37.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T06:28:37.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:28:37.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T06:28:37.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-06-03T06:28:37.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-06-03T06:28:37.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-06-03T06:28:37.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-06-03T06:28:37.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-06-03T06:28:37.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-06-03T06:28:37.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T06:28:37.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T06:28:37.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T06:28:37.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T06:28:37.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T06:28:37.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T06:28:38.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T06:28:38.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:28:38.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:28:38.273 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:28:38.273 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:28:38.274 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:28:38.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:28:38.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T06:28:38.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T06:28:38.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T06:28:38.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:28:38.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:28:38.459 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T06:28:38.459 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T06:28:38.460 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:28:38.460 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:28:38.460 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:28:38.460 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:28:38.460 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:28:38.460 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:28:38.460 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:28:38.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:28:38.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T06:28:38.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T06:28:38.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:28:38.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:28:38.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T06:28:38.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T06:28:38.544 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:28:38.544 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:28:38.544 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:28:38.544 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:28:38.545 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:28:38.545 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:28:38.545 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:28:38.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:28:38.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T06:28:38.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-06-03T06:28:38.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T06:28:38.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T06:28:38.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T06:28:38.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T06:28:38.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T06:28:38.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T06:28:39.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T06:28:39.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:28:39.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:28:39.452 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:28:39.452 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:28:39.452 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:28:39.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:28:39.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:28:39.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:28:39.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-06-03T06:28:39.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-03T06:28:39.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-06-03T06:28:39.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:28:39.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:28:39.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T06:28:39.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:28:39.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:28:39.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:28:39.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:28:39.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:28:39.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:28:39.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:28:39.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:28:39.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:28:39.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:28:39.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:28:39.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:28:39.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:28:39.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:28:39.488 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:28:39.488 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:28:39.488 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:28:39.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:28:39.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:28:39.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:28:39.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:28:39.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:28:39.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:28:39.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:28:39.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:28:39.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:28:39.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:28:39.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:28:39.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:28:39.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:28:39.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:28:39.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:28:39.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=68911f2a-c2fb-42f7-a8ea-af62e74ca453 2024-06-03T06:28:39.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 68911f2a-c2fb-42f7-a8ea-af62e74ca453' 2024-06-03T06:28:39.490 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 68911f2a-c2fb-42f7-a8ea-af62e74ca453 2024-06-03T06:28:39.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:28:39.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCXYl1mjzEiHhAA6QogtngukUP5bMmucxNR6A== 2024-06-03T06:28:39.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCXYl1mjzEiHhAA6QogtngukUP5bMmucxNR6A=="}' 2024-06-03T06:28:39.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 68911f2a-c2fb-42f7-a8ea-af62e74ca453 -i td/osd-scrub-repair/0/new.json 2024-06-03T06:28:39.758 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:28:39.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T06:28:39.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCXYl1mjzEiHhAA6QogtngukUP5bMmucxNR6A== --osd-uuid 68911f2a-c2fb-42f7-a8ea-af62e74ca453 2024-06-03T06:28:39.798 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:39.798+0000 7f935ba21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:28:39.834 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:39.834+0000 7f935ba21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:28:39.835 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:39.835+0000 7f935ba21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:28:39.836 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:39.835+0000 7f935ba21640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:28:39.836 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:39.835+0000 7f935ba21640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:28:39.836 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:39.835+0000 7f935ba21640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:28:39.926 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:39.925+0000 7f935ba21640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T06:28:45.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T06:28:45.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:28:45.413 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:28:45.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T06:28:45.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:28:45.790 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:28:45.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T06:28:45.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:28:45.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:28:45.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:28:45.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:28:45.818 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:45.817+0000 7f6a8240c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:28:45.826 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:45.826+0000 7f6a8240c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:28:45.827 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:45.827+0000 7f6a8240c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:28:46.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T06:28:46.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:28:46.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:28:46.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:28:46.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:28:46.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:46.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:28:46.082 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:28:46.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:46.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:28:46.347 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:46.347+0000 7f6a8240c640 -1 Falling back to public interface 2024-06-03T06:28:46.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:47.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:47.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:47.373 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:28:47.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:28:47.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:47.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:28:47.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:48.660 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:28:48.659+0000 7f6a8240c640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:28:48.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:48.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:48.675 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:28:48.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:28:48.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:48.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:28:49.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:50.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:50.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:50.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:28:50.121 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:28:50.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:50.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:28:50.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:51.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:51.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:51.431 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:28:51.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:28:51.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:51.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:28:51.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:52.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:52.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:52.730 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:28:52.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:28:52.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:52.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:28:53.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:54.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:54.038 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:28:54.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:54.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:28:54.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:54.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:28:54.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:55.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:55.385 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:28:55.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:55.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:28:55.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:55.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:28:55.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:56.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:56.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:56.691 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:28:56.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:28:56.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:56.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:28:56.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:57.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:57.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:57.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:28:57.993 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:28:57.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:57.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:28:58.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:28:59.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:28:59.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:28:59.422 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:28:59.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:28:59.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:28:59.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:28:59.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:00.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:00.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:00.724 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:29:00.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:29:00.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:00.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:29:01.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:02.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:02.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:02.443 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:29:02.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:29:02.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:02.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:29:02.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:03.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:03.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:03.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:29:03.871 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:29:03.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:03.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:29:04.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:05.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:05.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:05.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:29:05.515 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:29:05.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:05.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:29:05.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:06.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:06.876 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:29:06.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:06.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:29:06.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:06.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:29:07.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:08.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:08.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:08.604 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:29:08.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:29:08.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:08.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:29:09.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:10.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:10.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:10.349 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:29:10.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:29:10.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:10.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:29:10.678 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/196896640,v1:127.0.0.1:6803/196896640] [v2:127.0.0.1:6804/196896640,v1:127.0.0.1:6805/196896640] exists,up 68911f2a-c2fb-42f7-a8ea-af62e74ca453 2024-06-03T06:29:10.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:29:10.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:29:10.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:29:10.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-03T06:29:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-06-03T06:29:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:29:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:29:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T06:29:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:29:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:29:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:29:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:29:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:29:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:29:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:29:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:29:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:29:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:29:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:29:10.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:29:10.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:29:10.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:29:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:29:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:29:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:29:10.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:29:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:29:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:29:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:29:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:29:10.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:29:10.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:29:10.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:29:10.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:29:10.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:29:10.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:29:10.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:29:10.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:29:10.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:29:10.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:29:10.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a68c99d8-3111-4a07-aff9-9e8216641514 2024-06-03T06:29:10.685 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 a68c99d8-3111-4a07-aff9-9e8216641514 2024-06-03T06:29:10.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a68c99d8-3111-4a07-aff9-9e8216641514' 2024-06-03T06:29:10.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:29:10.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC2Yl1mIF2+KRAAFyfWehgsQVKCFw6IwlaLLg== 2024-06-03T06:29:10.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC2Yl1mIF2+KRAAFyfWehgsQVKCFw6IwlaLLg=="}' 2024-06-03T06:29:10.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a68c99d8-3111-4a07-aff9-9e8216641514 -i td/osd-scrub-repair/1/new.json 2024-06-03T06:29:11.035 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:29:11.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T06:29:11.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2Yl1mIF2+KRAAFyfWehgsQVKCFw6IwlaLLg== --osd-uuid a68c99d8-3111-4a07-aff9-9e8216641514 2024-06-03T06:29:11.077 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:11.077+0000 7f17dc463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:29:11.111 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:11.111+0000 7f17dc463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:29:11.112 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:11.112+0000 7f17dc463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:29:11.113 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:11.112+0000 7f17dc463640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:29:11.113 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:11.112+0000 7f17dc463640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:29:11.113 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:11.112+0000 7f17dc463640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:29:11.178 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:11.178+0000 7f17dc463640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T06:29:16.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T06:29:16.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:29:16.869 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:29:16.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T06:29:16.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:29:17.246 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:29:17.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T06:29:17.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:29:17.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:29:17.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:29:17.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:29:17.274 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:17.273+0000 7f6c0820d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:29:17.276 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:17.276+0000 7f6c0820d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:29:17.277 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:17.277+0000 7f6c0820d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:29:17.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T06:29:17.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:29:17.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:29:17.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:29:17.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:29:17.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:17.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:29:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:29:17.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:17.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:17.791 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:17.791+0000 7f6c0820d640 -1 Falling back to public interface 2024-06-03T06:29:17.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:18.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:18.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:18.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:29:18.829 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:29:18.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:18.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:19.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:20.113 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:20.112+0000 7f6c0820d640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:29:20.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:20.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:20.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:29:20.128 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:29:20.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:20.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:20.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:21.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:21.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:21.430 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:29:21.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:29:21.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:21.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:21.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:22.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:22.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:22.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:29:22.730 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:29:22.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:22.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:23.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:24.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:24.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:24.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:29:24.077 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:29:24.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:24.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:24.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:25.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:25.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:25.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:29:25.377 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:29:25.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:25.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:25.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:26.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:26.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:26.720 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:29:26.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:29:26.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:26.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:27.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:28.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:28.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:28.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:29:28.153 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:29:28.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:28.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:28.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:29.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:29.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:29.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:29:29.454 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:29:29.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:29.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:29.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:30.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:30.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:30.757 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:29:30.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:29:30.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:30.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:31.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:32.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:32.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:32.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:29:32.083 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:29:32.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:32.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:32.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:33.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:33.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:33.840 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:29:33.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:29:33.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:33.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:34.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:35.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:35.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:35.584 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:29:35.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:29:35.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:35.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:35.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:36.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:36.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:36.970 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:29:36.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:29:36.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:36.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:37.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:38.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:38.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:38.441 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:29:38.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:29:38.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:38.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:38.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:39.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:39.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:39.799 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:29:39.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:29:39.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:39.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:40.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:41.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:41.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:41.211 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:29:41.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:29:41.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:41.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:41.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:42.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:42.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:42.922 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:29:42.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:29:42.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:42.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:29:43.225 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3725570072,v1:127.0.0.1:6811/3725570072] [v2:127.0.0.1:6812/3725570072,v1:127.0.0.1:6813/3725570072] exists,up a68c99d8-3111-4a07-aff9-9e8216641514 2024-06-03T06:29:43.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:29:43.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:29:43.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:29:43.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-03T06:29:43.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-06-03T06:29:43.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:29:43.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:29:43.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T06:29:43.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:29:43.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:29:43.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:29:43.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:29:43.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:29:43.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:29:43.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:29:43.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:29:43.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:29:43.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:29:43.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:29:43.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:29:43.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:29:43.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:29:43.231 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:29:43.231 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:29:43.231 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:29:43.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:29:43.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:29:43.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:29:43.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:29:43.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:29:43.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:29:43.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:29:43.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:29:43.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:29:43.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:29:43.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:29:43.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:29:43.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:29:43.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:29:43.233 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:29:43.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bce7c1ac-8aba-4659-95f6-8cfab8bdbbec 2024-06-03T06:29:43.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bce7c1ac-8aba-4659-95f6-8cfab8bdbbec' 2024-06-03T06:29:43.233 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 bce7c1ac-8aba-4659-95f6-8cfab8bdbbec 2024-06-03T06:29:43.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:29:43.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDXYl1m0oa7DhAAeFvHbsrrvsVwZbCVaNMBtA== 2024-06-03T06:29:43.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDXYl1m0oa7DhAAeFvHbsrrvsVwZbCVaNMBtA=="}' 2024-06-03T06:29:43.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bce7c1ac-8aba-4659-95f6-8cfab8bdbbec -i td/osd-scrub-repair/2/new.json 2024-06-03T06:29:43.649 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:29:43.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-03T06:29:43.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXYl1m0oa7DhAAeFvHbsrrvsVwZbCVaNMBtA== --osd-uuid bce7c1ac-8aba-4659-95f6-8cfab8bdbbec 2024-06-03T06:29:43.691 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:43.691+0000 7f9a7b211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:29:43.734 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:43.734+0000 7f9a7b211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:29:43.735 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:43.735+0000 7f9a7b211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:29:43.736 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:43.735+0000 7f9a7b211640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:29:43.736 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:43.735+0000 7f9a7b211640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:29:43.736 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:43.735+0000 7f9a7b211640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:29:43.817 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:43.816+0000 7f9a7b211640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T06:29:49.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-03T06:29:49.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:29:49.897 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T06:29:49.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T06:29:49.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:29:50.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T06:29:50.284 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:29:50.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:29:50.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:29:50.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:29:50.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:29:50.312 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:50.311+0000 7f7c6a00e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:29:50.313 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:50.313+0000 7f7c6a00e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:29:50.315 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:50.314+0000 7f7c6a00e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:29:50.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T06:29:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:29:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:29:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:29:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:29:50.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:50.590 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:29:50.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:29:50.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:50.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:29:50.829 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:50.829+0000 7f7c6a00e640 -1 Falling back to public interface 2024-06-03T06:29:50.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:51.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:51.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:51.880 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:29:51.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:29:51.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:51.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:29:52.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:53.181 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:29:53.180+0000 7f7c6a00e640 -1 osd.2 0 log_to_monitors true 2024-06-03T06:29:53.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:53.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:53.185 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:29:53.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:29:53.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:53.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:29:53.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:54.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:54.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:54.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:29:54.506 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:29:54.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:54.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:29:54.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:55.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:55.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:55.826 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:29:55.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:29:55.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:55.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:29:56.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:57.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:57.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:57.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:29:57.231 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:29:57.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:57.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:29:57.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:58.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:58.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:58.535 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:29:58.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:29:58.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:58.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:29:58.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:29:59.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:29:59.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:29:59.860 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:29:59.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:29:59.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:29:59.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:30:00.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:01.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:01.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:01.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:30:01.217 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:30:01.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:01.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:30:01.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:02.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:02.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:02.519 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:30:02.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:30:02.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:02.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:30:02.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:03.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:03.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:03.822 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:30:03.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:30:03.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:03.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:30:04.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:05.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:05.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:05.214 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:30:05.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:30:05.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:05.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:30:05.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:06.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:06.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:06.605 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:30:06.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:30:06.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:06.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:30:07.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:08.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:08.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:08.238 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:30:08.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:30:08.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:08.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:30:08.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:09.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:09.696 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:30:09.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:09.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:30:09.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:09.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:30:10.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:11.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:11.099 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:30:11.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:11.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:30:11.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:11.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:30:11.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:12.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:12.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:12.793 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:30:12.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:30:12.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:12.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:30:13.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:14.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:14.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:14.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:30:14.376 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:30:14.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:14.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:30:14.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:15.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:15.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:15.677 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:30:15.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:30:15.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:15.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:30:16.017 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2519150709,v1:127.0.0.1:6819/2519150709] [v2:127.0.0.1:6820/2519150709,v1:127.0.0.1:6821/2519150709] exists,up bce7c1ac-8aba-4659-95f6-8cfab8bdbbec 2024-06-03T06:30:16.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:30:16.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:30:16.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:30:16.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-03T06:30:16.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-06-03T06:30:16.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:30:16.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:30:16.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-03T06:30:16.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:30:16.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:30:16.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:30:16.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:30:16.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:30:16.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:30:16.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:30:16.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:30:16.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:30:16.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:30:16.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:30:16.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:30:16.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:30:16.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:30:16.022 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:30:16.022 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:30:16.022 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:30:16.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:30:16.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:30:16.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:30:16.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:30:16.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:30:16.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:30:16.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:30:16.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:30:16.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:30:16.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:30:16.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:30:16.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:30:16.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:30:16.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:30:16.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:30:16.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c13f28c9-c6e3-4efc-acdf-7e5d713b78ec 2024-06-03T06:30:16.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 c13f28c9-c6e3-4efc-acdf-7e5d713b78ec' 2024-06-03T06:30:16.025 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 c13f28c9-c6e3-4efc-acdf-7e5d713b78ec 2024-06-03T06:30:16.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:30:16.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD4Yl1mY0FZAhAACGJraCIJJK9sHBSDCLWa0A== 2024-06-03T06:30:16.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD4Yl1mY0FZAhAACGJraCIJJK9sHBSDCLWa0A=="}' 2024-06-03T06:30:16.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c13f28c9-c6e3-4efc-acdf-7e5d713b78ec -i td/osd-scrub-repair/3/new.json 2024-06-03T06:30:16.381 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:30:16.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-06-03T06:30:16.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD4Yl1mY0FZAhAACGJraCIJJK9sHBSDCLWa0A== --osd-uuid c13f28c9-c6e3-4efc-acdf-7e5d713b78ec 2024-06-03T06:30:16.423 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:30:16.421+0000 7f6222f6e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:30:16.466 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:30:16.464+0000 7f6222f6e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:30:16.467 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:30:16.465+0000 7f6222f6e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:30:16.467 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:30:16.466+0000 7f6222f6e640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-03T06:30:16.467 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:30:16.466+0000 7f6222f6e640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-03T06:30:16.467 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:30:16.466+0000 7f6222f6e640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-03T06:30:16.541 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:30:16.540+0000 7f6222f6e640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-06-03T06:30:22.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-06-03T06:30:22.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:30:22.525 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-06-03T06:30:22.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-03T06:30:22.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:30:22.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-03T06:30:22.945 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-06-03T06:30:22.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:30:22.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:30:22.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:30:22.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:30:22.973 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:30:22.972+0000 7fc35cd44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:30:22.975 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:30:22.974+0000 7fc35cd44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:30:22.976 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:30:22.976+0000 7fc35cd44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:30:23.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-03T06:30:23.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:30:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:30:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:30:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:30:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:30:23.238 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:30:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:23.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:23.491 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:30:23.490+0000 7fc35cd44640 -1 Falling back to public interface 2024-06-03T06:30:23.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:24.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:24.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:24.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:30:24.531 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:30:24.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:24.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:24.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:25.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:25.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:25.838 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:30:25.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:30:25.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:25.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:25.875 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:30:25.874+0000 7fc35cd44640 -1 osd.3 0 log_to_monitors true 2024-06-03T06:30:26.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:27.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:27.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:27.243 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:30:27.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:30:27.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:27.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:27.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:28.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:28.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:28.556 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:30:28.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:30:28.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:28.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:28.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:29.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:29.981 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:30:29.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:29.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:30:29.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:29.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:30.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:31.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:31.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:31.282 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:30:31.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:30:31.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:31.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:31.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:32.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:32.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:32.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:30:32.582 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:30:32.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:32.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:33.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:34.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:34.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:34.056 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:30:34.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:30:34.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:34.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:34.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:35.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:35.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:35.360 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:30:35.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:30:35.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:35.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:35.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:36.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:36.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:36.663 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:30:36.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:30:36.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:36.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:37.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:38.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:38.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:38.056 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:30:38.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:30:38.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:38.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:38.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:39.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:39.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:39.355 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:30:39.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:30:39.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:39.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:39.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:40.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:40.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:40.884 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:30:40.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:30:40.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:40.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:41.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:42.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:42.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:42.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:30:42.486 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:30:42.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:42.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:43.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:44.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:44.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:44.268 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:30:44.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:30:44.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:44.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:44.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:45.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:45.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:45.624 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:30:45.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:30:45.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:45.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:46.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:47.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:47.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:47.662 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:30:47.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:30:47.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:47.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:48.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:30:49.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:30:49.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:30:49.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:30:49.433 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:30:49.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:30:49.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:30:49.734 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3691488381,v1:127.0.0.1:6827/3691488381] [v2:127.0.0.1:6828/3691488381,v1:127.0.0.1:6829/3691488381] exists,up c13f28c9-c6e3-4efc-acdf-7e5d713b78ec 2024-06-03T06:30:49.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:30:49.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:30:49.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:30:49.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-06-03T06:30:49.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T06:30:50.028 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-06-03T06:30:50.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T06:30:50.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T06:30:50.444 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T06:30:50.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:30:51.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T06:30:53.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-06-03T06:30:53.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:30:53.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:30:53.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:30:53.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:30:53.665 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:30:53.665 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:30:53.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:30:53.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:30:53.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:30:53.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:30:53.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:30:53.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:30:53.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:30:53.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:30:53.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:30:54.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:30:54.065 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:30:54.065 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:30:54.066 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-06-03T06:30:54.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:30:54.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:30:54.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:30:54.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-06-03T06:30:54.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-06-03T06:30:54.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-06-03T06:30:54.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:30:54.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:30:54.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-06-03T06:30:54.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-06-03T06:30:54.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672976' 2024-06-03T06:30:54.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:30:54.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:30:54.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-06-03T06:30:54.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-06-03T06:30:54.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672976 2-64424509456' 2024-06-03T06:30:54.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:30:54.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:30:54.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345931 2024-06-03T06:30:54.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345931 2024-06-03T06:30:54.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672976 2-64424509456 3-85899345931' 2024-06-03T06:30:54.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:30:54.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-06-03T06:30:54.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:30:54.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:30:54.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-06-03T06:30:54.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:30:54.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-06-03T06:30:54.537 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836509 2024-06-03T06:30:54.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-06-03T06:30:54.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:30:54.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-06-03T06:30:54.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:30:55.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:30:55.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:30:56.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-06-03T06:30:56.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:30:56.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-06-03T06:30:56.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:30:56.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:30:56.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-06-03T06:30:56.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:30:56.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-06-03T06:30:56.168 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672976 2024-06-03T06:30:56.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-06-03T06:30:56.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:30:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-06-03T06:30:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:30:56.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-06-03T06:30:56.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:30:56.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:30:56.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-06-03T06:30:56.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:30:56.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-06-03T06:30:56.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-06-03T06:30:56.503 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509456 2024-06-03T06:30:56.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:30:56.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-06-03T06:30:56.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:30:56.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345931 2024-06-03T06:30:56.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:30:56.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:30:56.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345931 2024-06-03T06:30:56.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:30:56.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345931 2024-06-03T06:30:56.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-06-03T06:30:56.802 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345931 2024-06-03T06:30:56.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:30:57.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345931 -lt 85899345931 2024-06-03T06:30:57.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:30:57.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:30:57.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:30:57.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T06:30:57.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:30:57.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:30:57.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:30:57.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:30:57.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:30:57.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:30:57.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:30:57.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:30:57.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:30:57.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:30:57.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:30:58.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T06:30:58.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:30:58.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:30:58.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-06-03T06:30:58.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-03T06:30:58.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-03T06:30:58.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-06-03T06:30:58.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-03T06:30:58.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-06-03T06:30:58.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-03T06:30:58.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-03T06:30:59.327 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'ecpool' created 2024-06-03T06:30:59.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:31:00.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-06-03T06:31:00.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-03T06:31:00.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:31:00.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:31:00.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:31:00.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:31:00.343 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:31:00.343 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:31:00.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:31:00.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:31:00.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:31:00.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:31:00.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:31:00.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:31:00.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:31:00.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:31:00.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:31:00.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:31:00.768 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:31:00.768 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:31:00.768 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-06-03T06:31:00.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:31:00.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:00.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:31:00.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836511 2024-06-03T06:31:00.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836511 2024-06-03T06:31:00.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511' 2024-06-03T06:31:00.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:00.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:31:00.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-06-03T06:31:00.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-06-03T06:31:00.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672978' 2024-06-03T06:31:00.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:00.984 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:31:01.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-06-03T06:31:01.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-06-03T06:31:01.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672978 2-64424509458' 2024-06-03T06:31:01.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:01.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:31:01.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345933 2024-06-03T06:31:01.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345933 2024-06-03T06:31:01.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672978 2-64424509458 3-85899345933' 2024-06-03T06:31:01.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:01.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836511 2024-06-03T06:31:01.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:01.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:31:01.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836511 2024-06-03T06:31:01.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:01.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836511 2024-06-03T06:31:01.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-06-03T06:31:01.203 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836511 2024-06-03T06:31:01.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:31:01.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836511 2024-06-03T06:31:01.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:31:02.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:31:02.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:31:02.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836511 2024-06-03T06:31:02.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:02.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-06-03T06:31:02.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:02.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:31:02.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-06-03T06:31:02.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:02.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-06-03T06:31:02.869 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672978 2024-06-03T06:31:02.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-06-03T06:31:02.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:31:03.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-06-03T06:31:03.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:03.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-06-03T06:31:03.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:03.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:31:03.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-06-03T06:31:03.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:03.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-06-03T06:31:03.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-06-03T06:31:03.163 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509458 2024-06-03T06:31:03.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:31:03.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-06-03T06:31:03.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:03.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345933 2024-06-03T06:31:03.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:03.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:31:03.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345933 2024-06-03T06:31:03.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:03.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345933 2024-06-03T06:31:03.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2024-06-03T06:31:03.457 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345933 2024-06-03T06:31:03.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:31:03.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345933 -lt 85899345933 2024-06-03T06:31:03.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:31:03.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:31:03.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:31:04.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:31:04.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:31:04.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:31:04.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:31:04.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:31:04.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:31:04.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:31:04.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:31:04.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:31:04.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:31:04.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:31:04.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:31:04.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-03T06:31:04.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-03T06:31:04.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:31:04.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-03T06:31:04.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:31:04.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:31:04.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:31:04.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:31:04.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:31:04.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:31:04.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:31:04.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:31:04.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:31:04.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:31:05.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:31:05.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:31:05.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:31:05.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:31:05.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-03T06:31:05.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-03T06:31:05.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:31:05.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:31:05.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:31:05.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:31:05.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:31:05.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:31:05.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:31:05.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T06:31:05.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T06:31:05.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T06:31:05.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:31:05.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T06:31:06.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:31:06.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:31:06.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:31:06.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:31:06.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:31:06.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:31:06.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:31:06.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:31:06.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:31:06.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:31:06.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:31:06.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:31:06.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:31:06.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:31:06.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:31:06.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-03T06:31:06.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-06-03T06:31:06.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-03T06:31:06.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-06-03T06:31:06.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-06-03T06:31:06.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T06:31:06.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T06:31:06.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-03T06:31:06.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T06:31:06.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T06:31:06.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T06:31:07.134 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T06:31:07.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T06:31:07.654 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T06:31:07.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T06:31:07.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T06:31:07.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-03T06:31:07.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-06-03T06:31:07.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-03T06:31:07.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:31:07.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:31:07.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:31:08.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-06-03T06:31:08.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-06-03T06:31:08.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-03T06:31:08.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-06-03T06:31:08.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-06-03T06:31:08.277 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:31:08.277 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T06:31:08.596 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-06-03T06:31:08.596 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:31:08.596 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:31:08.596 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-06-03T06:31:08.596 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-06-03T06:31:08.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-06-03T06:31:08.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-06-03T06:31:08.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-06-03T06:31:08.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-06-03T06:31:08.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-06-03T06:31:08.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-03T06:31:08.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-03T06:31:08.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-06-03T06:31:08.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:31:08.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:31:08.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:31:08.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:31:08.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:31:08.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:31:08.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:31:08.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:31:08.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:31:08.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:31:08.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:31:08.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:31:08.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:31:08.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:31:08.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:31:08.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:31:08.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:31:08.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:31:08.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:31:08.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:31:08.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:31:08.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:31:08.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:31:08.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-03T06:31:10.406 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-06-03T06:31:10.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:31:10.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:31:10.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:31:10.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:31:10.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:31:10.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:31:10.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:31:10.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:31:10.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:31:10.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:31:10.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:31:10.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:31:10.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:31:10.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:31:10.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:31:10.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:31:10.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:31:10.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:31:10.927 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:31:10.927 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:31:10.928 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:31:10.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:31:10.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:31:10.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:31:10.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:31:10.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:31:10.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:31:10.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:31:10.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:31:10.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:31:10.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:31:10.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:31:10.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:31:10.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:31:10.929 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.3 2024-06-03T06:31:10.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:31:10.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:31:10.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:31:10.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:31:10.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:31:10.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:31:10.958 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:10.956+0000 7ff3e961d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:31:10.960 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:10.959+0000 7ff3e961d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:31:10.962 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:10.960+0000 7ff3e961d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:31:11.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:31:11.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:31:11.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:31:11.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:31:11.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:31:11.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:31:11.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:31:11.232 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:31:11.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:31:11.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:31:11.476 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:11.475+0000 7ff3e961d640 -1 Falling back to public interface 2024-06-03T06:31:11.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:31:12.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:31:12.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:31:12.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:31:12.521 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:31:12.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:31:12.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:31:12.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:31:13.850 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:13.850+0000 7ff3e961d640 -1 osd.3 36 log_to_monitors true 2024-06-03T06:31:13.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:31:13.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:31:13.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:31:13.876 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:31:13.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:31:13.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:31:14.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:31:15.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:31:15.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:31:15.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:31:15.292 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:31:15.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:31:15.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:31:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:osd.3 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [20,36) [v2:127.0.0.1:6826/3870961060,v1:127.0.0.1:6827/3870961060] [v2:127.0.0.1:6828/3870961060,v1:127.0.0.1:6829/3870961060] exists,up c13f28c9-c6e3-4efc-acdf-7e5d713b78ec 2024-06-03T06:31:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:31:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:31:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:31:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:31:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:31:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:31:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:31:15.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:31:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:31:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:31:15.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:31:15.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:31:15.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:31:15.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:31:15.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:31:15.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:31:15.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:31:15.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:31:15.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:31:16.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:31:16.205 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:31:16.205 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:31:16.205 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-06-03T06:31:16.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:31:16.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:16.205 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:31:16.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-06-03T06:31:16.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-06-03T06:31:16.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-06-03T06:31:16.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:16.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:31:16.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-06-03T06:31:16.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-06-03T06:31:16.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672982' 2024-06-03T06:31:16.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:16.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:31:16.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-06-03T06:31:16.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-06-03T06:31:16.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672982 2-64424509462' 2024-06-03T06:31:16.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:16.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:31:16.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-06-03T06:31:16.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-06-03T06:31:16.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672982 2-64424509462 3-171798691842' 2024-06-03T06:31:16.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:16.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-06-03T06:31:16.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:16.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:31:16.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-06-03T06:31:16.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:16.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-06-03T06:31:16.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-06-03T06:31:16.678 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836516 2024-06-03T06:31:16.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:31:16.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836516 2024-06-03T06:31:16.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:31:17.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:31:17.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:31:18.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836516 2024-06-03T06:31:18.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:18.269 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-06-03T06:31:18.269 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:18.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:31:18.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-06-03T06:31:18.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:18.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-06-03T06:31:18.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-06-03T06:31:18.272 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672982 2024-06-03T06:31:18.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:31:18.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-06-03T06:31:18.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:18.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-06-03T06:31:18.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:18.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:31:18.622 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-06-03T06:31:18.622 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:18.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-06-03T06:31:18.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-06-03T06:31:18.624 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509462 2024-06-03T06:31:18.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:31:18.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509462 2024-06-03T06:31:18.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:18.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-171798691842 2024-06-03T06:31:18.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:18.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:31:18.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-171798691842 2024-06-03T06:31:18.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:18.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-06-03T06:31:18.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 171798691842' 2024-06-03T06:31:18.915 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 171798691842 2024-06-03T06:31:18.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:31:19.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-06-03T06:31:19.205 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:31:19.205 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:31:19.205 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:31:19.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:31:19.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:31:19.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:31:19.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:31:19.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:31:19.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:31:19.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:31:19.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:31:19.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:31:19.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:31:19.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:31:19.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:31:20.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:31:20.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:31:20.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:31:20.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-03T06:31:20.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:31:20.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:31:20.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:31:20.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:31:20.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-03T06:31:20.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-03T06:31:20.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:31:20.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:31:20.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:31:20.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:31:20.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:31:20.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:31:20.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:30:59.209000+0000 2024-06-03T06:31:20.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:31:21.066 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:31:21.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:30:59.209000+0000 2024-06-03T06:31:21.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:31:21.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:30:59.209000+0000 2024-06-03T06:31:21.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:31:21.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:31:21.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:31:21.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:31:21.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:31:21.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:31:21.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:31:21.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:31:21.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:30:59.209000+0000 '>' 2024-06-03T06:30:59.209000+0000 2024-06-03T06:31:21.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:31:22.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:31:22.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:31:22.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:31:22.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:31:22.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:31:22.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:31:22.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:31:22.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:30:59.209000+0000 '>' 2024-06-03T06:30:59.209000+0000 2024-06-03T06:31:22.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:31:23.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:31:23.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:31:23.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:31:23.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:31:23.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:31:23.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:31:23.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:31:23.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:30:59.209000+0000 '>' 2024-06-03T06:30:59.209000+0000 2024-06-03T06:31:23.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:31:24.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:31:24.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:31:24.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:31:24.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:31:24.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:31:24.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:31:24.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:31:25.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:30:59.209000+0000 '>' 2024-06-03T06:30:59.209000+0000 2024-06-03T06:31:25.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:31:26.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:31:26.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:31:26.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:31:26.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:31:26.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:31:26.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:31:26.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:31:26.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:31:21.893080+0000 '>' 2024-06-03T06:30:59.209000+0000 2024-06-03T06:31:26.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:31:26.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:31:26.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:31:26.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:31:26.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:31:26.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:31:26.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:31:26.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:31:26.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:31:26.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:31:26.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:31:26.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:31:26.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:31:26.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:31:26.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:31:26.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:31:26.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:31:26.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:31:26.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:31:26.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:31:26.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:31:26.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:31:26.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:31:26.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:31:26.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-03T06:31:28.386 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-03T06:31:28.386 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:31:28.386 INFO:tasks.workunit.client.0.smithi039.stdout:hinfo_key 2024-06-03T06:31:28.386 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:31:28.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:31:28.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:31:28.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:31:28.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:31:28.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:31:28.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:31:28.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:31:28.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:31:28.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:31:28.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:31:28.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:31:28.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:31:28.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:31:28.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:31:28.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:31:28.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:31:28.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:31:28.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:31:28.662 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:31:28.662 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:31:28.662 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:31:28.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:31:28.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:31:28.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:31:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:31:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:31:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:31:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:31:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:31:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:31:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:31:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:31:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:31:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:31:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.3 2024-06-03T06:31:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:31:28.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:31:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:31:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:31:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:31:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:31:28.692 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:28.692+0000 7f504dca2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:31:28.695 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:28.694+0000 7f504dca2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:31:28.696 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:28.696+0000 7f504dca2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:31:28.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:31:28.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:31:28.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:31:28.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:31:28.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:31:28.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:31:28.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:31:28.965 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:31:28.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:31:28.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:31:29.222 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:29.222+0000 7f504dca2640 -1 Falling back to public interface 2024-06-03T06:31:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:31:30.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:31:30.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:31:30.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:31:30.255 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:31:30.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:31:30.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:31:30.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:31:31.522 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:31.522+0000 7f504dca2640 -1 osd.3 42 log_to_monitors true 2024-06-03T06:31:31.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:31:31.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:31:31.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:31:31.555 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:31:31.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:31:31.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:31:31.994 INFO:tasks.workunit.client.0.smithi039.stderr:osd.3 up in weight 1 up_from 46 up_thru 40 down_at 43 last_clean_interval [40,42) [v2:127.0.0.1:6826/1768115258,v1:127.0.0.1:6827/1768115258] [v2:127.0.0.1:6828/1768115258,v1:127.0.0.1:6829/1768115258] exists,up c13f28c9-c6e3-4efc-acdf-7e5d713b78ec 2024-06-03T06:31:31.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:31:31.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:31:31.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:31:31.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:31:31.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:31:31.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:31:31.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:31:31.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:31:31.995 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:31:31.995 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:31:31.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:31:31.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:31:31.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:31:32.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:31:32.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:31:32.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:31:32.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:31:32.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:31:32.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:31:32.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:31:32.437 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:31:32.437 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:31:32.437 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-06-03T06:31:32.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:31:32.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:32.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:31:32.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836520 2024-06-03T06:31:32.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836520 2024-06-03T06:31:32.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520' 2024-06-03T06:31:32.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:32.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:31:32.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-06-03T06:31:32.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-06-03T06:31:32.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-42949672986' 2024-06-03T06:31:32.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:32.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:31:32.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-06-03T06:31:32.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-06-03T06:31:32.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-42949672986 2-64424509467' 2024-06-03T06:31:32.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:32.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:31:33.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495618 2024-06-03T06:31:33.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495618 2024-06-03T06:31:33.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-42949672986 2-64424509467 3-197568495618' 2024-06-03T06:31:33.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:33.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836520 2024-06-03T06:31:33.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:33.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:31:33.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836520 2024-06-03T06:31:33.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:33.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836520 2024-06-03T06:31:33.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-06-03T06:31:33.029 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836520 2024-06-03T06:31:33.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:31:33.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836520 2024-06-03T06:31:33.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:31:34.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:31:34.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:31:34.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836520 2024-06-03T06:31:34.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:34.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-06-03T06:31:34.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:34.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:31:34.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-06-03T06:31:34.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:34.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-06-03T06:31:34.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-06-03T06:31:34.740 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672986 2024-06-03T06:31:34.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:31:35.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672986 2024-06-03T06:31:35.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:35.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-06-03T06:31:35.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:35.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:31:35.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-06-03T06:31:35.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:35.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-06-03T06:31:35.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-06-03T06:31:35.031 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509467 2024-06-03T06:31:35.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:31:35.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-06-03T06:31:35.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:35.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495618 2024-06-03T06:31:35.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:35.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:31:35.325 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495618 2024-06-03T06:31:35.325 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:35.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495618 2024-06-03T06:31:35.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495618' 2024-06-03T06:31:35.327 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 197568495618 2024-06-03T06:31:35.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:31:35.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495618 -lt 197568495618 2024-06-03T06:31:35.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:31:35.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:31:35.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:31:36.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:31:36.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:31:36.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:31:36.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:31:36.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:31:36.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:31:36.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:31:36.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:31:36.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-03T06:31:36.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:31:36.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:31:36.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:31:36.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-03T06:31:36.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-03T06:31:36.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:31:36.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-03T06:31:36.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:31:36.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:31:36.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:31:36.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:31:36.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:31:36.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:31:36.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:31:36.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:31:36.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:31:36.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:31:37.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-03T06:31:37.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:31:37.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:31:37.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:31:37.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-03T06:31:37.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-06-03T06:31:37.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:31:37.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:31:37.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:31:37.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:31:37.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:31:37.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:31:37.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:31:37.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T06:31:37.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T06:31:37.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T06:31:37.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:31:37.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T06:31:37.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:31:37.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:31:37.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:31:37.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:31:37.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:31:37.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:31:37.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:31:37.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:31:38.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:31:38.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:31:38.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:31:38.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:31:38.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:31:38.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:31:38.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:31:38.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:31:38.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:31:38.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-06-03T06:31:38.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-03T06:31:38.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-03T06:31:38.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-06-03T06:31:38.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:31:38.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:31:38.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:31:38.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:31:38.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:31:38.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:31:38.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:31:38.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:31:38.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:31:38.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:31:38.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:31:38.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:31:38.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:31:38.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:31:38.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:31:38.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:31:39.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:31:39.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:31:39.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:31:39.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:31:39.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:31:39.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:31:39.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:31:39.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-03T06:31:40.862 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-03T06:31:41.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:31:41.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:31:41.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:31:41.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:31:41.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:31:41.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:31:41.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:31:41.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:31:41.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:31:41.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:31:41.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:31:41.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:31:41.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:31:41.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:31:41.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:31:41.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:31:41.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:31:41.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:31:41.389 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:31:41.389 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:31:41.389 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:31:41.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:31:41.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:31:41.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:31:41.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:31:41.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:31:41.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:31:41.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:31:41.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:31:41.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:31:41.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:31:41.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:31:41.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:31:41.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:31:41.391 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:31:41.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:31:41.391 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:31:41.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:31:41.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:31:41.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:31:41.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:31:41.420 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:41.419+0000 7efcaa807640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:31:41.422 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:41.421+0000 7efcaa807640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:31:41.423 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:41.423+0000 7efcaa807640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:31:41.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:31:41.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:31:41.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:31:41.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:31:41.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:31:41.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:31:41.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:31:41.694 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:31:41.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:31:41.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:31:41.939 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:41.939+0000 7efcaa807640 -1 Falling back to public interface 2024-06-03T06:31:42.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:31:43.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:31:43.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:31:43.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:31:43.024 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:31:43.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:31:43.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:31:43.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:31:44.271 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:44.271+0000 7efcaa807640 -1 osd.1 48 log_to_monitors true 2024-06-03T06:31:44.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:31:44.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:31:44.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:31:44.328 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:31:44.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:31:44.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:31:44.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:31:45.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:31:45.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:31:45.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:31:45.664 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:31:45.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:31:45.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:31:46.031 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 49 last_clean_interval [10,48) [v2:127.0.0.1:6810/2716669250,v1:127.0.0.1:6811/2716669250] [v2:127.0.0.1:6812/2716669250,v1:127.0.0.1:6813/2716669250] exists,up a68c99d8-3111-4a07-aff9-9e8216641514 2024-06-03T06:31:46.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:31:46.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:31:46.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:31:46.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:31:46.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:31:46.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:31:46.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:31:46.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:31:46.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:31:46.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:31:46.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:31:46.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:31:46.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:31:46.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:31:46.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:31:46.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:31:46.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:31:46.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:31:46.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:31:46.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:31:46.465 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:31:46.465 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:31:46.465 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-06-03T06:31:46.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:31:46.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:46.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:31:46.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836524 2024-06-03T06:31:46.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836524 2024-06-03T06:31:46.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524' 2024-06-03T06:31:46.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:46.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:31:46.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266690 2024-06-03T06:31:46.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266690 2024-06-03T06:31:46.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-227633266690' 2024-06-03T06:31:46.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:46.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:31:46.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509471 2024-06-03T06:31:46.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509471 2024-06-03T06:31:46.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-227633266690 2-64424509471' 2024-06-03T06:31:46.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:46.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:31:46.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495622 2024-06-03T06:31:46.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495622 2024-06-03T06:31:46.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-227633266690 2-64424509471 3-197568495622' 2024-06-03T06:31:46.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:46.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836524 2024-06-03T06:31:46.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:46.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:31:46.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836524 2024-06-03T06:31:46.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:46.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836524 2024-06-03T06:31:46.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-06-03T06:31:46.915 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836524 2024-06-03T06:31:46.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:31:47.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836522 -lt 21474836524 2024-06-03T06:31:47.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:31:48.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:31:48.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:31:48.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836524 -lt 21474836524 2024-06-03T06:31:48.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:48.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266690 2024-06-03T06:31:48.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:48.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:31:48.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266690 2024-06-03T06:31:48.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:48.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266690 2024-06-03T06:31:48.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266690' 2024-06-03T06:31:48.506 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 227633266690 2024-06-03T06:31:48.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:31:48.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266690 -lt 227633266690 2024-06-03T06:31:48.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:48.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509471 2024-06-03T06:31:48.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:48.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:31:48.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509471 2024-06-03T06:31:48.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:48.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509471 2024-06-03T06:31:48.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-06-03T06:31:48.806 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509471 2024-06-03T06:31:48.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:31:49.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509471 -lt 64424509471 2024-06-03T06:31:49.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:31:49.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495622 2024-06-03T06:31:49.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:31:49.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:31:49.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495622 2024-06-03T06:31:49.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:31:49.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495622 2024-06-03T06:31:49.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495622' 2024-06-03T06:31:49.133 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 197568495622 2024-06-03T06:31:49.133 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:31:49.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495622 -lt 197568495622 2024-06-03T06:31:49.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:31:49.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:31:49.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:31:49.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:31:49.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:31:49.789 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:31:49.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:31:49.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:31:49.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:31:49.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:31:49.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:31:50.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:31:50.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:31:50.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:31:50.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:31:50.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:31:50.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:31:50.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:31:50.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-03T06:31:50.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:31:50.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:31:50.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:31:50.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:31:50.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-03T06:31:50.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-03T06:31:50.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:31:50.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:31:50.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:31:50.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:31:50.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:31:50.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:31:51.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:31:21.893080+0000 2024-06-03T06:31:51.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:31:51.310 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:31:51.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:31:21.893080+0000 2024-06-03T06:31:51.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:31:51.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:31:21.893080+0000 2024-06-03T06:31:51.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:31:51.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:31:51.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:31:51.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:31:51.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:31:51.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:31:51.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:31:51.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:31:51.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:31:21.893080+0000 '>' 2024-06-03T06:31:21.893080+0000 2024-06-03T06:31:51.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:31:52.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:31:52.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:31:52.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:31:52.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:31:52.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:31:52.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:31:52.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:31:52.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:31:21.893080+0000 '>' 2024-06-03T06:31:21.893080+0000 2024-06-03T06:31:52.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:31:53.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:31:53.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:31:53.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:31:53.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:31:53.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:31:53.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:31:53.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:31:54.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:31:51.505355+0000 '>' 2024-06-03T06:31:21.893080+0000 2024-06-03T06:31:54.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:31:54.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:31:54.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:31:54.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:31:54.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:31:54.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:31:54.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:31:54.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:31:54.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:31:54.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:31:54.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:31:54.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:31:54.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:31:54.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:31:54.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:31:54.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:31:54.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:31:54.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:31:54.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:31:54.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:31:54.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:31:54.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:31:54.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:31:54.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:31:54.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-03T06:31:55.957 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-03T06:31:55.957 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:31:55.957 INFO:tasks.workunit.client.0.smithi039.stdout:hinfo_key 2024-06-03T06:31:55.957 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:31:56.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:31:56.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:31:56.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:31:56.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:31:56.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:31:56.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:31:56.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:31:56.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:31:56.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:31:56.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:31:56.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:31:56.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:31:56.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:31:56.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:31:56.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:31:56.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:31:56.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:31:56.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:31:56.226 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:31:56.226 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:31:56.226 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:31:56.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:31:56.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:31:56.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:31:56.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:31:56.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:31:56.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:31:56.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:31:56.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:31:56.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:31:56.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:31:56.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:31:56.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:31:56.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:31:56.228 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.3 2024-06-03T06:31:56.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:31:56.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:31:56.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:31:56.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:31:56.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:31:56.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:31:56.256 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:56.255+0000 7f8d4000e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:31:56.258 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:56.258+0000 7f8d4000e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:31:56.259 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:56.259+0000 7f8d4000e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:31:56.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:31:56.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:31:56.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:31:56.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:31:56.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:31:56.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:31:56.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:31:56.524 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:31:56.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:31:56.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:31:56.774 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:56.773+0000 7f8d4000e640 -1 Falling back to public interface 2024-06-03T06:31:56.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:31:57.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:31:57.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:31:57.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:31:57.813 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:31:57.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:31:57.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:31:58.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:31:58.966 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:31:58.964+0000 7f8d4000e640 -1 osd.3 55 log_to_monitors true 2024-06-03T06:31:59.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:31:59.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:31:59.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:31:59.117 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:31:59.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:31:59.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:31:59.484 INFO:tasks.workunit.client.0.smithi039.stderr:osd.3 up in weight 1 up_from 59 up_thru 53 down_at 56 last_clean_interval [46,55) [v2:127.0.0.1:6826/3537233854,v1:127.0.0.1:6827/3537233854] [v2:127.0.0.1:6828/3537233854,v1:127.0.0.1:6829/3537233854] exists,up c13f28c9-c6e3-4efc-acdf-7e5d713b78ec 2024-06-03T06:31:59.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:31:59.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:31:59.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:31:59.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:31:59.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:31:59.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:31:59.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:31:59.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:31:59.485 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:31:59.485 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:31:59.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:31:59.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:31:59.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:31:59.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:31:59.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:31:59.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:31:59.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:31:59.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:31:59.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:31:59.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:31:59.979 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:31:59.979 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:31:59.979 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-06-03T06:31:59.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:31:59.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:31:59.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:32:00.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836527 2024-06-03T06:32:00.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836527 2024-06-03T06:32:00.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527' 2024-06-03T06:32:00.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:00.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:32:00.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266694 2024-06-03T06:32:00.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266694 2024-06-03T06:32:00.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-227633266694' 2024-06-03T06:32:00.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:00.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:32:00.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509474 2024-06-03T06:32:00.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509474 2024-06-03T06:32:00.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-227633266694 2-64424509474' 2024-06-03T06:32:00.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:00.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:32:00.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070466 2024-06-03T06:32:00.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070466 2024-06-03T06:32:00.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-227633266694 2-64424509474 3-253403070466' 2024-06-03T06:32:00.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:00.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836527 2024-06-03T06:32:00.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:00.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:32:00.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836527 2024-06-03T06:32:00.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:00.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836527 2024-06-03T06:32:00.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-06-03T06:32:00.620 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836527 2024-06-03T06:32:00.620 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:32:00.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836526 -lt 21474836527 2024-06-03T06:32:00.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:32:01.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:32:01.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:32:02.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836527 2024-06-03T06:32:02.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:02.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266694 2024-06-03T06:32:02.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:02.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:32:02.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266694 2024-06-03T06:32:02.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:02.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266694 2024-06-03T06:32:02.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266694' 2024-06-03T06:32:02.218 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 227633266694 2024-06-03T06:32:02.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:32:02.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266694 -lt 227633266694 2024-06-03T06:32:02.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:02.507 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509474 2024-06-03T06:32:02.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:02.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:32:02.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509474 2024-06-03T06:32:02.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:02.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509474 2024-06-03T06:32:02.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-06-03T06:32:02.510 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509474 2024-06-03T06:32:02.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:32:02.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509474 -lt 64424509474 2024-06-03T06:32:02.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:02.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070466 2024-06-03T06:32:02.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:02.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:32:02.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070466 2024-06-03T06:32:02.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:02.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070466 2024-06-03T06:32:02.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070466' 2024-06-03T06:32:02.844 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 253403070466 2024-06-03T06:32:02.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:32:03.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070466 -lt 253403070466 2024-06-03T06:32:03.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:32:03.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:03.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:03.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:32:03.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:32:03.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:32:03.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:32:03.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:32:03.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:32:03.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:32:03.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:32:03.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:32:03.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:32:03.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:03.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:04.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:32:04.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:32:04.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:32:04.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:32:04.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:32:04.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-06-03T06:32:04.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-03T06:32:04.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-03T06:32:04.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-06-03T06:32:04.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-06-03T06:32:04.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-03T06:32:04.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:32:04.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:32:04.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:32:04.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 165886"' 2024-06-03T06:32:04.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 165886' 2024-06-03T06:32:04.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-06-03T06:32:04.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:32:04.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:32:04.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 165888"' 2024-06-03T06:32:04.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:32:04.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 165888' 2024-06-03T06:32:04.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-03T06:32:04.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 165886 165888' 2024-06-03T06:32:04.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:32:04.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:32:04.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 165886 2024-06-03T06:32:04.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/165889: /' 2024-06-03T06:32:04.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-06-03T06:32:04.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/165891: /' 2024-06-03T06:32:07.084 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:32:07.084 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:32:07.084 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:32:07.084 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:32:07.085 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:32:07.085 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:32:07.085 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:32:07.085 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:32:07.085 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:32:07.085 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:32:07.085 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:32:07.085 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:32:07.085 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:32:07.085 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:32:07.085 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:32:07.085 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:32:07.086 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:32:07.086 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:32:07.086 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:32:07.086 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:32:07.086 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:32:07.086 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:32:07.086 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-03T06:32:07.086 INFO:tasks.workunit.client.0.smithi039.stderr:165889: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-03T06:32:07.086 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:32:07.086 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:32:07.086 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:32:07.086 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:32:07.087 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:32:07.087 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:32:07.087 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:32:07.087 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:32:07.088 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:32:07.088 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:32:07.088 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:32:07.088 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:32:07.088 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:32:07.088 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:32:07.088 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:32:07.088 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:32:07.088 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:32:07.088 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:32:07.089 INFO:tasks.workunit.client.0.smithi039.stderr:165889: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:32:07.089 INFO:tasks.workunit.client.0.smithi039.stderr:165889: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:32:07.089 INFO:tasks.workunit.client.0.smithi039.stderr:165889: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:32:07.089 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:32:07.089 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:32:07.089 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:32:07.089 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:32:07.089 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:32:07.089 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:32:07.089 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:32:07.089 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:32:07.089 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:32:07.089 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:32:07.089 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:32:07.090 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:32:07.090 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:32:07.090 INFO:tasks.workunit.client.0.smithi039.stderr:165889: start osd.1 2024-06-03T06:32:07.090 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:32:07.266 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:32:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:32:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-03T06:32:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:32:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-06-03T06:32:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:32:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:32:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-03T06:32:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:32:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-03T06:32:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:32:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:32:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:32:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:32:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:32:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:32:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-06-03T06:32:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:32:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:32:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-03T06:32:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:32:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:32:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-06-03T06:32:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:165891: remove 2#2:eb822e21:::SOMETHING:head# 2024-06-03T06:32:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-03T06:32:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:32:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:32:07.269 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-03T06:32:07.269 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:32:07.269 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:32:07.269 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:32:07.269 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:32:07.269 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:32:07.269 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:32:07.269 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:32:07.269 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:32:07.269 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:32:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:32:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:32:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:32:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:32:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:32:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:165891: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:32:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:165891: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:32:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:165891: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:32:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:32:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:32:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:32:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:32:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:32:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:32:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:32:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:32:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:32:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:32:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:32:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:32:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-03T06:32:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:165891: start osd.2 2024-06-03T06:32:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:32:11.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:32:11.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:32:11.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:32:11.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:32:11.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:32:11.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: 2024-06-03T06:32:07.118+0000 7f246361d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:11.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: 2024-06-03T06:32:07.121+0000 7f246361d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:11.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: 2024-06-03T06:32:07.122+0000 7f246361d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:11.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:32:11.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:32:11.781 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:32:11.781 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:32:11.781 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:32:11.781 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:11.781 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:32:11.781 INFO:tasks.workunit.client.0.smithi039.stderr:165889: 0 2024-06-03T06:32:11.781 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:11.781 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:32:11.781 INFO:tasks.workunit.client.0.smithi039.stderr:165889: 2024-06-03T06:32:07.645+0000 7f246361d640 -1 Falling back to public interface 2024-06-03T06:32:11.781 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:11.781 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:11.781 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:11.781 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:32:11.782 INFO:tasks.workunit.client.0.smithi039.stderr:165889: 1 2024-06-03T06:32:11.782 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:11.782 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:32:11.782 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:11.782 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:11.782 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:11.782 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:32:11.782 INFO:tasks.workunit.client.0.smithi039.stderr:165889: 2 2024-06-03T06:32:11.782 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:11.782 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:32:11.782 INFO:tasks.workunit.client.0.smithi039.stderr:165889: 2024-06-03T06:32:10.222+0000 7f246361d640 -1 osd.1 60 log_to_monitors true 2024-06-03T06:32:11.782 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:11.782 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:11.783 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:11.783 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:32:11.783 INFO:tasks.workunit.client.0.smithi039.stderr:165889: 3 2024-06-03T06:32:11.783 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:11.783 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:32:11.834 INFO:tasks.workunit.client.0.smithi039.stderr:165889: osd.1 up in weight 1 up_from 65 up_thru 65 down_at 61 last_165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-03T06:32:11.834 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-03T06:32:11.834 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:32:11.834 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:32:11.834 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:32:11.834 INFO:tasks.workunit.client.0.smithi039.stderr:165891: 2024-06-03T06:32:07.296+0000 7f9ff4615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:11.834 INFO:tasks.workunit.client.0.smithi039.stderr:165891: 2024-06-03T06:32:07.298+0000 7f9ff4615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:11.834 INFO:tasks.workunit.client.0.smithi039.stderr:165891: 2024-06-03T06:32:07.299+0000 7f9ff4615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:11.834 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-03T06:32:11.834 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:32:11.835 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:32:11.835 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:32:11.835 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:32:11.835 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:11.835 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:32:11.835 INFO:tasks.workunit.client.0.smithi039.stderr:165891: 0 2024-06-03T06:32:11.835 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:11.835 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:32:11.835 INFO:tasks.workunit.client.0.smithi039.stderr:165891: 2024-06-03T06:32:07.820+0000 7f9ff4615640 -1 Falling back to public interface 2024-06-03T06:32:11.835 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:11.835 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:11.835 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:11.835 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:32:11.835 INFO:tasks.workunit.client.0.smithi039.stderr:165891: 1 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: 2 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: 2024-06-03T06:32:10.768+0000 7f9ff4615640 -1 osd.2 60 log_to_monitors true 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: 3 2024-06-03T06:32:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:11.837 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:32:12.549 INFO:tasks.workunit.client.0.smithi039.stderr:165891: osd.2 up in weight 1 up_from 66 up_thru 49 down_at 61 last_clean_interval [53,60) [v2:127.0.0.1:6810/940114491,v1:127.0.0.1:6811/940114491] [v2:127.0.0.1:6812/940114491,v1:127.0.0.1:6813/940114491] exists,up a68c99d8-3111-4a07-aff9-9e8216641514 2024-06-03T06:32:12.549 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:32:12.549 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:32:12.550 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:32:12.550 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:32:12.550 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:32:12.550 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:32:12.550 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:32:12.551 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:32:12.551 INFO:tasks.workunit.client.0.smithi039.stderr:165889: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:32:12.551 INFO:tasks.workunit.client.0.smithi039.stderr:165889: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:32:12.551 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:32:12.551 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:32:12.551 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:32:12.551 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:32:12.551 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:32:12.551 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:32:12.551 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:32:12.551 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:32:12.551 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:32:12.551 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:32:12.552 INFO:tasks.workunit.client.0.smithi039.stderr:165889: 1 2024-06-03T06:32:12.552 INFO:tasks.workunit.client.0.smithi039.stderr:165889: 2 2024-06-03T06:32:12.552 INFO:tasks.workunit.client.0.smithi039.stderr:165889: 3' 2024-06-03T06:32:12.552 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:32:12.552 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:12.552 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:32:12.552 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836531 2024-06-03T06:32:12.552 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836531 2024-06-03T06:32:12.552 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531' 2024-06-03T06:32:12.553 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:12.553 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:32:12.553 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874242 2024-06-03T06:32:12.553 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874242 2024-06-03T06:32:12.553 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-279172874242' 2024-06-03T06:32:12.553 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:12.844 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tellclean_interval [15,60) [v2:127.0.0.1:6818/3567278231,v1:127.0.0.1:6819/3567278231] [v2:127.0.0.1:6820/3567278231,v1:127.0.0.1:6821/3567278231] exists,up bce7c1ac-8aba-4659-95f6-8cfab8bdbbec 2024-06-03T06:32:12.844 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:32:12.844 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:32:12.845 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:32:12.845 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:32:12.845 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:32:12.845 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:32:12.845 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:32:12.845 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:32:12.845 INFO:tasks.workunit.client.0.smithi039.stderr:165891: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:32:12.845 INFO:tasks.workunit.client.0.smithi039.stderr:165891: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:32:12.845 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:32:12.845 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:32:12.845 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:32:12.846 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:32:12.846 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:32:12.846 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:32:12.846 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:32:12.846 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:32:12.846 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:32:12.846 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:32:12.846 INFO:tasks.workunit.client.0.smithi039.stderr:165891: 1 2024-06-03T06:32:12.846 INFO:tasks.workunit.client.0.smithi039.stderr:165891: 2 2024-06-03T06:32:12.847 INFO:tasks.workunit.client.0.smithi039.stderr:165891: 3' 2024-06-03T06:32:12.847 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:32:12.847 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:12.847 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:32:12.847 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836532 2024-06-03T06:32:12.847 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836532 2024-06-03T06:32:12.847 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532' 2024-06-03T06:32:12.847 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:12.848 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:32:12.848 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874243 2024-06-03T06:32:12.848 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874243 2024-06-03T06:32:12.848 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-279172874243' 2024-06-03T06:32:12.848 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:14.777 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph osd.2 flush_pg_stats 2024-06-03T06:32:14.777 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-06-03T06:32:14.777 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-06-03T06:32:14.777 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-279172874242 2-283467841538' 2024-06-03T06:32:14.778 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:14.778 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:32:14.778 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070469 2024-06-03T06:32:14.778 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070469 2024-06-03T06:32:14.778 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-279172874242 2-283467841538 3-253403070469' 2024-06-03T06:32:14.778 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:14.778 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836531 2024-06-03T06:32:14.778 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:14.778 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:32:14.779 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836531 2024-06-03T06:32:14.779 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:14.779 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836531 2024-06-03T06:32:14.779 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-06-03T06:32:14.779 INFO:tasks.workunit.client.0.smithi039.stderr:165889: waiting osd.0 seq 21474836531 2024-06-03T06:32:14.779 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:32:14.779 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836531 2024-06-03T06:32:14.779 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:32:14.779 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:32:14.779 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:32:14.779 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836531 2024-06-03T06:32:14.779 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:14.779 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874242 2024-06-03T06:32:14.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:14.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:32:14.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874242 2024-06-03T06:32:14.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:14.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874242 2024-06-03T06:32:14.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874242' 2024-06-03T06:32:14.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: waiting osd.1 seq 279172874242 2024-06-03T06:32:14.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:32:14.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874243 -lt 279172874242 2024-06-03T06:32:14.780 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:15.156 INFO:tasks.workunit.client.0.smithi039.stderr:165889: tell osd.2 flush_pg_stats 2024-06-03T06:32:15.156 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841539 2024-06-03T06:32:15.156 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841539 2024-06-03T06:32:15.156 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-279172874243 2-283467841539' 2024-06-03T06:32:15.156 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:15.156 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:32:15.156 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070470 2024-06-03T06:32:15.156 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070470 2024-06-03T06:32:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-279172874243 2-283467841539 3-253403070470' 2024-06-03T06:32:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836532 2024-06-03T06:32:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:32:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836532 2024-06-03T06:32:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836532 2024-06-03T06:32:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2024-06-03T06:32:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:165891: waiting osd.0 seq 21474836532 2024-06-03T06:32:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:32:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836532 2024-06-03T06:32:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:32:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:32:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:32:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836532 2024-06-03T06:32:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874243 2024-06-03T06:32:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:32:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874243 2024-06-03T06:32:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874243 2024-06-03T06:32:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874243' 2024-06-03T06:32:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:165891: waiting osd.1 seq 279172874243 2024-06-03T06:32:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:32:15.158 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874243 -lt 279172874243 2024-06-03T06:32:15.159 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:16.067 INFO:tasks.workunit.client.0.smithi039.stderr:1658//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-283467841538 2024-06-03T06:32:16.067 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:16.067 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:32:16.067 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-283467841538 2024-06-03T06:32:16.067 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:16.067 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-06-03T06:32:16.067 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 283467841538' 2024-06-03T06:32:16.067 INFO:tasks.workunit.client.0.smithi039.stderr:165889: waiting osd.2 seq 283467841538 2024-06-03T06:32:16.068 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:32:16.068 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841538 2024-06-03T06:32:16.068 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:16.068 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070469 2024-06-03T06:32:16.068 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:16.068 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:32:16.068 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070469 2024-06-03T06:32:16.068 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:16.068 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070469 2024-06-03T06:32:16.068 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070469' 2024-06-03T06:32:16.068 INFO:tasks.workunit.client.0.smithi039.stderr:165889: waiting osd.3 seq 253403070469 2024-06-03T06:32:16.068 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:32:16.068 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070470 -lt 253403070469 2024-06-03T06:32:16.069 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:32:16.069 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:16.069 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:16.069 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:32:16.069 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:32:16.069 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:32:16.069 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:32:16.069 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:32:16.069 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:32:16.069 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:32:16.069 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:32:16.070 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-03T06:32:16.070 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:32:16.471 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.clien91: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-283467841539 2024-06-03T06:32:16.471 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:16.471 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:32:16.472 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-283467841539 2024-06-03T06:32:16.472 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:16.472 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841539 2024-06-03T06:32:16.472 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 283467841539' 2024-06-03T06:32:16.472 INFO:tasks.workunit.client.0.smithi039.stderr:165891: waiting osd.2 seq 283467841539 2024-06-03T06:32:16.472 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:32:16.473 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841539 2024-06-03T06:32:16.473 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:16.473 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070470 2024-06-03T06:32:16.473 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:16.473 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:32:16.474 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070470 2024-06-03T06:32:16.474 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:16.474 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070470 2024-06-03T06:32:16.474 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070470' 2024-06-03T06:32:16.474 INFO:tasks.workunit.client.0.smithi039.stderr:165891: waiting osd.3 seq 253403070470 2024-06-03T06:32:16.475 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:32:16.475 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070471 -lt 253403070470 2024-06-03T06:32:16.475 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:32:16.475 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:16.475 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:16.475 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:32:16.476 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:32:16.476 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:32:16.476 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:32:16.476 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:32:16.476 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:32:16.476 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:32:16.477 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:32:16.477 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-03T06:32:16.477 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:32:17.628 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.ct.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:17.628 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:17.629 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-03T06:32:17.629 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-03T06:32:17.629 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:32:17.629 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-03T06:32:17.629 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:32:17.629 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:32:17.629 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:32:17.629 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:32:17.629 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:32:17.629 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:32:17.629 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:32:17.629 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:32:17.630 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:32:17.630 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:32:17.630 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-03T06:32:17.630 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:32:17.630 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:17.630 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:17.630 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-03T06:32:17.630 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-06-03T06:32:17.630 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:32:17.630 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:32:17.630 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:32:17.631 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:32:17.631 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:32:17.631 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:32:17.631 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:32:17.631 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T06:32:17.631 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T06:32:18.102 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clonlient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:18.102 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:18.102 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-03T06:32:18.103 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-03T06:32:18.103 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:32:18.103 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-03T06:32:18.103 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:32:18.103 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:32:18.103 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:32:18.103 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:32:18.103 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:32:18.103 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:32:18.103 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:32:18.103 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:32:18.103 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:32:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:32:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-03T06:32:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:32:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-03T06:32:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-06-03T06:32:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:32:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:32:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:32:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:32:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:32:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:32:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:32:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T06:32:18.105 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T06:32:18.521 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/e.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T06:32:18.521 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:32:18.521 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T06:32:18.521 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:32:18.521 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:32:18.522 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:32:18.522 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:32:18.522 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:32:18.522 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:32:18.522 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:32:18.522 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:32:18.522 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:32:18.522 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:32:18.522 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:18.522 INFO:tasks.workunit.client.0.smithi039.stderr:165889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:18.523 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:32:18.523 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:32:18.523 INFO:tasks.workunit.client.0.smithi039.stderr:165889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:32:18.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:32:18.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:32:18.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 165888 2024-06-03T06:32:18.958 INFO:tasks.workunit.client.0.smithi039.stderr:clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T06:32:18.959 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:32:18.959 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T06:32:18.959 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:32:18.959 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:32:18.959 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:32:18.959 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:32:18.959 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:32:18.959 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:32:18.959 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:32:18.959 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:32:18.959 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:32:18.959 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:32:18.960 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:18.960 INFO:tasks.workunit.client.0.smithi039.stderr:165891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:18.960 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:32:18.960 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:32:18.960 INFO:tasks.workunit.client.0.smithi039.stderr:165891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:32:18.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:32:18.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:32:18.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:32:18.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:32:18.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-03T06:32:18.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:32:18.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-03T06:32:18.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:32:18.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:32:18.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:32:18.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:32:19.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-03T06:32:19.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-03T06:32:19.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:32:19.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:32:19.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:32:19.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:32:19.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:32:19.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:32:19.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:31:51.505355+0000 2024-06-03T06:32:19.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:32:19.815 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:32:19.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:31:51.505355+0000 2024-06-03T06:32:19.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:32:19.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:31:51.505355+0000 2024-06-03T06:32:19.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:32:19.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:32:19.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:32:19.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:32:19.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:32:19.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:32:19.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:32:19.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:32:20.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:31:51.505355+0000 '>' 2024-06-03T06:31:51.505355+0000 2024-06-03T06:32:20.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:32:21.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:32:21.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:32:21.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:32:21.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:32:21.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:32:21.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:32:21.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:32:21.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:31:51.505355+0000 '>' 2024-06-03T06:31:51.505355+0000 2024-06-03T06:32:21.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:32:22.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:32:22.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:32:22.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:32:22.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:32:22.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:32:22.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:32:22.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:32:22.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:31:51.505355+0000 '>' 2024-06-03T06:31:51.505355+0000 2024-06-03T06:32:22.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:32:23.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:32:23.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:32:23.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:32:23.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:32:23.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:32:23.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:32:23.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:32:24.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:31:51.505355+0000 '>' 2024-06-03T06:31:51.505355+0000 2024-06-03T06:32:24.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:32:25.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:32:25.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:32:25.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:32:25.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:32:25.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:32:25.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:32:25.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:32:25.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:31:51.505355+0000 '>' 2024-06-03T06:31:51.505355+0000 2024-06-03T06:32:25.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:32:26.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:32:26.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:32:26.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:32:26.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:32:26.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:32:26.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:32:26.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:32:26.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:32:20.201148+0000 '>' 2024-06-03T06:31:51.505355+0000 2024-06-03T06:32:26.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:32:26.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-03T06:32:26.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:32:26.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:32:26.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:32:26.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 169007"' 2024-06-03T06:32:26.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 169007' 2024-06-03T06:32:26.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-03T06:32:26.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:32:26.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:32:26.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 169009"' 2024-06-03T06:32:26.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:32:26.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 169009' 2024-06-03T06:32:26.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-03T06:32:26.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/169010: /' 2024-06-03T06:32:26.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 169007 169009' 2024-06-03T06:32:26.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:32:26.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:32:26.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 169007 2024-06-03T06:32:26.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-03T06:32:26.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/169012: /' 2024-06-03T06:32:29.216 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:32:29.216 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:32:29.216 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-03T06:32:29.216 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:32:29.217 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-03T06:32:29.217 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:32:29.217 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:32:29.217 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-03T06:32:29.217 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:32:29.217 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-03T06:32:29.217 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:32:29.217 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:32:29.217 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:32:29.217 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:32:29.217 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:32:29.217 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:32:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-03T06:32:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:32:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:32:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-03T06:32:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:32:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:32:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-06-03T06:32:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:169012: _ 2024-06-03T06:32:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:169012: hinfo_key 2024-06-03T06:32:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:169012: snapset 2024-06-03T06:32:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-03T06:32:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:32:29.219 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:32:29.219 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-03T06:32:29.219 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:32:29.219 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:32:29.219 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:32:29.219 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:32:29.219 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:32:29.219 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:32:29.220 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:32:29.220 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:32:29.220 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:32:29.220 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:32:29.220 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:32:29.220 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:32:29.220 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:32:29.220 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:32:29.220 INFO:tasks.workunit.client.0.smithi039.stderr:169012: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:32:29.220 INFO:tasks.workunit.client.0.smithi039.stderr:169012: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:32:29.220 INFO:tasks.workunit.client.0.smithi039.stderr:169012: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:32:29.220 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:32:29.220 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:32:29.220 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:32:29.221 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:32:29.221 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:32:29.221 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:32:29.221 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:32:29.221 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:32:29.221 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:32:29.221 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:32:29.221 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:32:29.221 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:32:29.221 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-03T06:32:29.221 INFO:tasks.workunit.client.0.smithi039.stderr:169012: start osd.2 2024-06-03T06:32:29.248 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:32:29.248 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:32:29.248 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:32:29.248 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:32:29.248 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:32:29.248 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:32:29.249 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:32:29.249 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:32:29.249 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:32:29.249 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:32:29.249 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:32:29.249 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:32:29.249 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:32:29.249 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:32:29.249 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:32:29.249 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:32:29.249 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:32:29.249 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:32:29.249 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:32:29.250 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:32:29.250 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:32:29.250 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:32:29.250 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-03T06:32:29.250 INFO:tasks.workunit.client.0.smithi039.stderr:169010: _ 2024-06-03T06:32:29.250 INFO:tasks.workunit.client.0.smithi039.stderr:169010: hinfo_key 2024-06-03T06:32:29.250 INFO:tasks.workunit.client.0.smithi039.stderr:169010: snapset 2024-06-03T06:32:29.250 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:32:29.250 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:32:29.250 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:32:29.250 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:32:29.250 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:32:29.250 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:32:29.251 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:32:29.251 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:32:29.251 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:32:29.251 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:32:29.251 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:32:29.251 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:32:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:32:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:32:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:32:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:32:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:32:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:32:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:169010: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:32:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:169010: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:32:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:169010: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:32:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:32:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:32:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:32:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:32:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:32:29.253 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:32:29.253 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:32:29.253 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:32:29.253 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:32:29.253 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:32:29.253 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:32:29.253 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:32:29.253 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:32:29.253 INFO:tasks.workunit.client.0.smithi039.stderr:169010: start osd.1 2024-06-03T06:32:34.007 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:32:34.008 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:32:34.008 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:32:34.008 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:32:34.008 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:32:34.008 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:32:34.009 INFO:tasks.workunit.client.0.smithi039.stderr:169010: 2024-06-03T06:32:29.281+0000 7f1ae4e1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:34.009 INFO:tasks.workunit.client.0.smithi039.stderr:169010: 2024-06-03T06:32:29.284+0000 7f1ae4e1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:34.009 INFO:tasks.workunit.client.0.smithi039.stderr:169010: 2024-06-03T06:32:29.285+0000 7f1ae4e1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:34.009 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:32:34.009 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:32:34.009 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:32:34.010 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:32:34.010 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:32:34.010 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:34.010 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:32:34.010 INFO:tasks.workunit.client.0.smithi039.stderr:169010: 0 2024-06-03T06:32:34.011 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:34.011 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:32:34.011 INFO:tasks.workunit.client.0.smithi039.stderr:169010: 2024-06-03T06:32:29.808+0000 7f1ae4e1a640 -1 Falling back to public interface 2024-06-03T06:32:34.011 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:34.011 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:34.011 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:34.011 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:32:34.012 INFO:tasks.workunit.client.0.smithi039.stderr:169010: 1 2024-06-03T06:32:34.012 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:34.012 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:32:34.012 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:34.012 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:34.012 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:34.013 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:32:34.013 INFO:tasks.workunit.client.0.smithi039.stderr:169010: 2 2024-06-03T06:32:34.013 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:34.013 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:32:34.013 INFO:tasks.workunit.client.0.smithi039.stderr:169010: 2024-06-03T06:32:32.492+0000 7f1ae4e1a640 -1 osd.1 68 log_to_monitors true 2024-06-03T06:32:34.013 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:34.013 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:34.014 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:34.014 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:32:34.014 INFO:tasks.workunit.client.0.smithi039.stderr:169010: 3 2024-06-03T06:32:34.014 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:34.014 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:32:34.015 INFO:tasks.workunit.client.0.smithi039.stderr:169010: osd.1 up in weight 1 up_from 74 up_thru 74 down_aovery_ops 2024-06-03T06:32:34.015 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-03T06:32:34.015 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-03T06:32:34.015 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:32:34.016 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:32:34.016 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:32:34.016 INFO:tasks.workunit.client.0.smithi039.stderr:169012: 2024-06-03T06:32:29.252+0000 7f952cd25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:34.016 INFO:tasks.workunit.client.0.smithi039.stderr:169012: 2024-06-03T06:32:29.255+0000 7f952cd25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:34.016 INFO:tasks.workunit.client.0.smithi039.stderr:169012: 2024-06-03T06:32:29.257+0000 7f952cd25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:34.016 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-03T06:32:34.017 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:32:34.017 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:32:34.017 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:32:34.017 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:32:34.017 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:34.017 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:32:34.018 INFO:tasks.workunit.client.0.smithi039.stderr:169012: 0 2024-06-03T06:32:34.018 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:34.018 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:32:34.018 INFO:tasks.workunit.client.0.smithi039.stderr:169012: 2024-06-03T06:32:29.772+0000 7f952cd25640 -1 Falling back to public interface 2024-06-03T06:32:34.018 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:34.019 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:34.019 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:34.019 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:32:34.019 INFO:tasks.workunit.client.0.smithi039.stderr:169012: 1 2024-06-03T06:32:34.019 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:34.019 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:32:34.019 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:34.020 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:34.020 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:34.020 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:32:34.020 INFO:tasks.workunit.client.0.smithi039.stderr:169012: 2 2024-06-03T06:32:34.020 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:34.020 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:32:34.021 INFO:tasks.workunit.client.0.smithi039.stderr:169012: 2024-06-03T06:32:32.350+0000 7f952cd25640 -1 osd.2 68 log_to_monitors true 2024-06-03T06:32:34.021 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:34.021 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:34.021 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:34.021 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:32:34.021 INFO:tasks.workunit.client.0.smithi039.stderr:169012: 3 2024-06-03T06:32:34.022 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:34.022 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:32:34.955 INFO:tasks.workunit.client.0.smithi039.stderr:169012: osd.2 up in weight 1 up_from 74 up_thru 49 down_at 69 last_clean_interval [65,68) [v2:127.0.0.1:6818/1324177416,v1:127.0.0.1:6819/1324177416] [v2:127.0.0.1:6820/1324177416,v1:127.0.0.1:6821/1324177416] exists,up a68c99d8-3111-4a07-aff9-9e8216641514 2024-06-03T06:32:34.956 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:32:34.956 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:32:34.956 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:32:34.956 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:32:34.956 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:32:34.956 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:32:34.956 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:32:34.956 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:32:34.957 INFO:tasks.workunit.client.0.smithi039.stderr:169010: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:32:34.957 INFO:tasks.workunit.client.0.smithi039.stderr:169010: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:32:34.957 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:32:34.957 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:32:34.957 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:32:34.957 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:32:34.957 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:32:34.957 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:32:34.957 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:32:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:32:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:32:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:32:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:169010: 1 2024-06-03T06:32:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:169010: 2 2024-06-03T06:32:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:169010: 3' 2024-06-03T06:32:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:32:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:32:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836538 2024-06-03T06:32:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836538 2024-06-03T06:32:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538' 2024-06-03T06:32:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:32:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-06-03T06:32:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-06-03T06:32:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-317827579906' 2024-06-03T06:32:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stat 69 last_clean_interval [66,68) [v2:127.0.0.1:6810/497358648,v1:127.0.0.1:6811/497358648] [v2:127.0.0.1:6812/497358648,v1:127.0.0.1:6813/497358648] exists,up bce7c1ac-8aba-4659-95f6-8cfab8bdbbec 2024-06-03T06:32:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:32:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:32:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:32:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:32:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:32:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:32:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:32:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:32:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:169012: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:32:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:169012: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:32:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:32:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:32:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:32:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:32:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:32:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:32:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:32:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:32:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:32:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:32:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:169012: 1 2024-06-03T06:32:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:169012: 2 2024-06-03T06:32:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:169012: 3' 2024-06-03T06:32:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:32:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:32:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836537 2024-06-03T06:32:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836537 2024-06-03T06:32:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537' 2024-06-03T06:32:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:32:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579907 2024-06-03T06:32:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579907 2024-06-03T06:32:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-317827579907' 2024-06-03T06:32:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:37.090 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ts: ceph tell osd.2 flush_pg_stats 2024-06-03T06:32:37.091 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-06-03T06:32:37.091 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-06-03T06:32:37.091 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-317827579906 2-317827579906' 2024-06-03T06:32:37.091 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:37.091 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:32:37.091 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070477 2024-06-03T06:32:37.091 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070477 2024-06-03T06:32:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-317827579906 2-317827579906 3-253403070477' 2024-06-03T06:32:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836538 2024-06-03T06:32:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:32:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836538 2024-06-03T06:32:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836538 2024-06-03T06:32:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-06-03T06:32:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:169010: waiting osd.0 seq 21474836538 2024-06-03T06:32:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:32:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836538 2024-06-03T06:32:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:32:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:32:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:32:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836538 2024-06-03T06:32:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-317827579906 2024-06-03T06:32:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:32:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-317827579906 2024-06-03T06:32:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-06-03T06:32:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 317827579906' 2024-06-03T06:32:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:169010: waiting osd.1 seq 317827579906 2024-06-03T06:32:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:32:37.094 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579906 2024-06-03T06:32:37.208 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in ceph tell osd.2 flush_pg_stats 2024-06-03T06:32:37.209 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579907 2024-06-03T06:32:37.209 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579907 2024-06-03T06:32:37.209 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-317827579907 2-317827579907' 2024-06-03T06:32:37.209 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:37.209 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:32:37.209 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070476 2024-06-03T06:32:37.209 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070476 2024-06-03T06:32:37.209 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-317827579907 2-317827579907 3-253403070476' 2024-06-03T06:32:37.209 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:37.209 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836537 2024-06-03T06:32:37.209 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:37.209 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:32:37.209 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836537 2024-06-03T06:32:37.210 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:37.210 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836537 2024-06-03T06:32:37.210 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-06-03T06:32:37.210 INFO:tasks.workunit.client.0.smithi039.stderr:169012: waiting osd.0 seq 21474836537 2024-06-03T06:32:37.210 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:32:37.210 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836537 2024-06-03T06:32:37.210 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:32:37.210 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:32:37.210 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:32:37.210 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836537 2024-06-03T06:32:37.210 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:37.210 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-317827579907 2024-06-03T06:32:37.210 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:37.211 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:32:37.211 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-317827579907 2024-06-03T06:32:37.211 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:37.211 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579907 2024-06-03T06:32:37.211 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 317827579907' 2024-06-03T06:32:37.211 INFO:tasks.workunit.client.0.smithi039.stderr:169012: waiting osd.1 seq 317827579907 2024-06-03T06:32:37.211 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:32:37.211 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579907 2024-06-03T06:32:38.428 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seq$seqs 2024-06-03T06:32:38.429 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579906 2024-06-03T06:32:38.429 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:38.429 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:32:38.429 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579906 2024-06-03T06:32:38.429 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:38.429 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-06-03T06:32:38.429 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-06-03T06:32:38.429 INFO:tasks.workunit.client.0.smithi039.stderr:169010: waiting osd.2 seq 317827579906 2024-06-03T06:32:38.429 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:32:38.429 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579906 2024-06-03T06:32:38.429 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:38.430 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070477 2024-06-03T06:32:38.430 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:38.430 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:32:38.430 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070477 2024-06-03T06:32:38.430 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:38.430 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070477 2024-06-03T06:32:38.430 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070477' 2024-06-03T06:32:38.430 INFO:tasks.workunit.client.0.smithi039.stderr:169010: waiting osd.3 seq 253403070477 2024-06-03T06:32:38.431 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:32:38.431 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070477 -lt 253403070477 2024-06-03T06:32:38.431 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:32:38.431 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:38.431 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:38.431 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:32:38.431 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:32:38.431 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:32:38.431 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:32:38.431 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:32:38.431 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:32:38.431 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:32:38.431 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:32:38.432 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:32:38.432 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:32:38.573 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtes 2024-06-03T06:32:38.573 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579907 2024-06-03T06:32:38.573 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:38.573 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:32:38.573 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579907 2024-06-03T06:32:38.573 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:38.573 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579907 2024-06-03T06:32:38.573 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579907' 2024-06-03T06:32:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:169012: waiting osd.2 seq 317827579907 2024-06-03T06:32:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:32:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579907 2024-06-03T06:32:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070476 2024-06-03T06:32:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:32:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070476 2024-06-03T06:32:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070476 2024-06-03T06:32:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070476' 2024-06-03T06:32:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:169012: waiting osd.3 seq 253403070476 2024-06-03T06:32:38.575 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:32:38.575 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070477 -lt 253403070476 2024-06-03T06:32:38.575 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:32:38.575 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:38.575 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:38.575 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:32:38.575 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:32:38.575 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:32:38.575 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:32:38.575 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:32:38.575 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:32:38.575 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:32:38.575 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:32:38.576 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:32:38.576 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:32:38.800 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/cst/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:38.800 INFO:tasks.workunit.client.0.smithi039.stderr:169010: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:32:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:32:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:169010: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:32:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:32:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:32:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 169009 2024-06-03T06:32:38.950 INFO:tasks.workunit.client.0.smithi039.stderr:lone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:38.950 INFO:tasks.workunit.client.0.smithi039.stderr:169012: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:38.950 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:32:38.950 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:32:38.950 INFO:tasks.workunit.client.0.smithi039.stderr:169012: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:32:38.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:32:38.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:32:38.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:32:38.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:32:38.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-03T06:32:38.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:32:38.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:32:38.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:32:38.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-06-03T06:32:38.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-03T06:32:38.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-03T06:32:38.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-06-03T06:32:38.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-06-03T06:32:38.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-03T06:32:38.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:32:38.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:32:38.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:32:38.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 171686"' 2024-06-03T06:32:38.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 171686' 2024-06-03T06:32:38.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:32:38.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:32:38.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:32:38.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:32:38.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 171688"' 2024-06-03T06:32:38.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 171688' 2024-06-03T06:32:38.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-03T06:32:38.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 171686 171688' 2024-06-03T06:32:38.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:32:38.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:32:38.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/171689: /' 2024-06-03T06:32:38.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 171686 2024-06-03T06:32:38.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:32:38.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/171691: /' 2024-06-03T06:32:41.637 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:32:41.637 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:32:41.637 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:32:41.637 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:32:41.637 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:32:41.637 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:32:41.638 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:32:41.638 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:32:41.638 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:32:41.638 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:32:41.638 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:32:41.639 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:32:41.639 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:32:41.639 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:32:41.639 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:32:41.639 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:32:41.639 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:32:41.639 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:32:41.639 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:32:41.639 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:32:41.639 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:32:41.640 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:32:41.640 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-03T06:32:41.640 INFO:tasks.workunit.client.0.smithi039.stderr:171691: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-03T06:32:41.640 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:32:41.640 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:32:41.640 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:32:41.640 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:32:41.640 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:32:41.640 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:32:41.640 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:32:41.640 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:32:41.641 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:32:41.641 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:32:41.641 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:32:41.641 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:32:41.641 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:32:41.641 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:32:41.641 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:32:41.641 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:32:41.641 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:32:41.642 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:32:41.642 INFO:tasks.workunit.client.0.smithi039.stderr:171691: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:32:41.642 INFO:tasks.workunit.client.0.smithi039.stderr:171691: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:32:41.642 INFO:tasks.workunit.client.0.smithi039.stderr:171691: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:32:41.642 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:32:41.642 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:32:41.642 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:32:41.642 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:32:41.642 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:32:41.643 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:32:41.643 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:32:41.643 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:32:41.643 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:32:41.643 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:32:41.643 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:32:41.643 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:32:41.644 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:32:41.644 INFO:tasks.workunit.client.0.smithi039.stderr:171691: start osd.1 2024-06-03T06:32:41.644 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:32:42.153 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:32:42.153 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:32:42.153 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:32:42.153 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:32:42.153 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:32:42.153 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:32:42.154 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:32:42.154 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:32:42.154 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:32:42.154 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:32:42.154 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:32:42.154 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:32:42.154 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:32:42.154 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:32:42.154 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:32:42.154 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:32:42.154 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:32:42.154 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:32:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:32:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:32:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:32:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:32:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-03T06:32:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:171689: remove 0#2:eb822e21:::SOMETHING:head# 2024-06-03T06:32:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:32:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:32:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:32:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:32:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:32:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:32:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:32:42.156 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:32:42.156 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:32:42.156 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:32:42.156 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:32:42.156 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:32:42.156 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:32:42.156 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:32:42.156 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:32:42.156 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:32:42.157 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:32:42.157 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:32:42.157 INFO:tasks.workunit.client.0.smithi039.stderr:171689: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:32:42.157 INFO:tasks.workunit.client.0.smithi039.stderr:171689: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:32:42.157 INFO:tasks.workunit.client.0.smithi039.stderr:171689: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:32:42.157 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:32:42.157 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:32:42.157 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:32:42.157 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:32:42.157 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:32:42.157 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:32:42.157 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:32:42.157 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:32:42.157 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:32:42.158 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:32:42.158 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:32:42.158 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:32:42.158 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:32:42.158 INFO:tasks.workunit.client.0.smithi039.stderr:171689: start osd.3 2024-06-03T06:32:42.158 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:32:46.243 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:32:46.243 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:32:46.243 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:32:46.243 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:32:46.243 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:32:46.243 INFO:tasks.workunit.client.0.smithi039.stderr:171691: 2024-06-03T06:32:41.671+0000 7f1ac181d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:46.243 INFO:tasks.workunit.client.0.smithi039.stderr:171691: 2024-06-03T06:32:41.699+0000 7f1ac181d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:46.243 INFO:tasks.workunit.client.0.smithi039.stderr:171691: 2024-06-03T06:32:41.700+0000 7f1ac181d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:46.243 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:32:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:32:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:32:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:32:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:32:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:32:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:171691: 0 2024-06-03T06:32:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:32:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:171691: 2024-06-03T06:32:42.221+0000 7f1ac181d640 -1 Falling back to public interface 2024-06-03T06:32:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:32:46.245 INFO:tasks.workunit.client.0.smithi039.stderr:171691: 1 2024-06-03T06:32:46.245 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:46.245 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:32:46.245 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:46.245 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:46.245 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:46.245 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:32:46.245 INFO:tasks.workunit.client.0.smithi039.stderr:171691: 2 2024-06-03T06:32:46.245 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:46.245 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:32:46.245 INFO:tasks.workunit.client.0.smithi039.stderr:171691: 2024-06-03T06:32:44.723+0000 7f1ac181d640 -1 osd.1 76 log_to_monitors true 2024-06-03T06:32:46.245 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:46.245 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:46.246 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:46.246 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:32:46.246 INFO:tasks.workunit.client.0.smithi039.stderr:171691: 3 2024-06-03T06:32:46.246 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:46.246 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:32:47.058 INFO:tasks.workunit.client.0.smithi039.stderr:171691: osd.1 up in weight 1 up_from 81 up_thru 81 down_at 77 last_171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:32:47.058 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:32:47.058 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:32:47.058 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:32:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:32:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:171689: 2024-06-03T06:32:42.182+0000 7ffa591a7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:171689: 2024-06-03T06:32:42.185+0000 7ffa591a7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:171689: 2024-06-03T06:32:42.186+0000 7ffa591a7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:32:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:32:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:32:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:32:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:32:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:32:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:32:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:171689: 0 2024-06-03T06:32:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:32:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:47.060 INFO:tasks.workunit.client.0.smithi039.stderr:171689: 2024-06-03T06:32:42.703+0000 7ffa591a7640 -1 Falling back to public interface 2024-06-03T06:32:47.060 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:47.060 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:47.060 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:47.060 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:32:47.060 INFO:tasks.workunit.client.0.smithi039.stderr:171689: 1 2024-06-03T06:32:47.060 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:47.060 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:32:47.060 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:47.060 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:47.060 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:47.060 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:32:47.060 INFO:tasks.workunit.client.0.smithi039.stderr:171689: 2 2024-06-03T06:32:47.060 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:47.061 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:32:47.061 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:32:47.061 INFO:tasks.workunit.client.0.smithi039.stderr:171689: 2024-06-03T06:32:45.537+0000 7ffa591a7640 -1 osd.3 76 log_to_monitors true 2024-06-03T06:32:47.061 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:32:47.061 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:32:47.061 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:32:47.061 INFO:tasks.workunit.client.0.smithi039.stderr:171689: 3 2024-06-03T06:32:47.061 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:32:47.061 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:32:47.275 INFO:tasks.workunit.client.0.smithi039.stderr:171689: osd.3 up in weight 1 up_from 84 up_thru 74 down_at 77 last_clean_interval [74,76) [v2:127.0.0.1:6818/3728988787,v1:127.0.0.1:6819/3728988787] [v2:127.0.0.1:6820/3728988787,v1:127.0.0.1:6821/3728988787] exists,up a68c99d8-3111-4a07-aff9-9e8216641514 2024-06-03T06:32:47.275 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:32:47.275 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:32:47.276 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:32:47.276 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:32:47.276 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:32:47.276 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:32:47.276 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:32:47.276 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:32:47.276 INFO:tasks.workunit.client.0.smithi039.stderr:171691: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:32:47.276 INFO:tasks.workunit.client.0.smithi039.stderr:171691: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:32:47.276 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:32:47.276 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:32:47.276 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:32:47.276 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:32:47.276 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:32:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:32:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:32:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:32:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:32:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:32:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:171691: 1 2024-06-03T06:32:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:171691: 2 2024-06-03T06:32:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:171691: 3' 2024-06-03T06:32:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:32:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:32:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836542 2024-06-03T06:32:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836542 2024-06-03T06:32:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542' 2024-06-03T06:32:47.277 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:47.278 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:32:47.278 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350978 2024-06-03T06:32:47.278 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350978 2024-06-03T06:32:47.278 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-347892350978' 2024-06-03T06:32:47.278 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:47.815 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [59,76) [v2:127.0.0.1:6826/3020951568,v1:127.0.0.1:6827/3020951568] [v2:127.0.0.1:6828/3020951568,v1:127.0.0.1:6829/3020951568] exists,up c13f28c9-c6e3-4efc-acdf-7e5d713b78ec 2024-06-03T06:32:47.815 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:32:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:32:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:32:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:32:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:32:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:32:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:32:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:32:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:171689: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:32:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:171689: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:32:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:32:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:32:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:32:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:32:47.817 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:32:47.817 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:32:47.817 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:32:47.817 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:32:47.817 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:32:47.817 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:32:47.817 INFO:tasks.workunit.client.0.smithi039.stderr:171689: 1 2024-06-03T06:32:47.817 INFO:tasks.workunit.client.0.smithi039.stderr:171689: 2 2024-06-03T06:32:47.817 INFO:tasks.workunit.client.0.smithi039.stderr:171689: 3' 2024-06-03T06:32:47.817 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:32:47.817 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:47.817 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:32:47.818 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836543 2024-06-03T06:32:47.818 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836543 2024-06-03T06:32:47.818 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543' 2024-06-03T06:32:47.818 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:47.818 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:32:47.818 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350979 2024-06-03T06:32:47.818 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350979 2024-06-03T06:32:47.818 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-347892350979' 2024-06-03T06:32:47.818 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:49.617 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:32:49.617 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579910 2024-06-03T06:32:49.617 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579910 2024-06-03T06:32:49.617 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-347892350978 2-317827579910' 2024-06-03T06:32:49.618 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:49.618 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:32:49.618 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-06-03T06:32:49.618 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-06-03T06:32:49.618 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-347892350978 2-317827579910 3-360777252866' 2024-06-03T06:32:49.618 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:49.618 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836542 2024-06-03T06:32:49.618 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:49.618 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:32:49.618 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836542 2024-06-03T06:32:49.618 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:49.618 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836542 2024-06-03T06:32:49.618 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2024-06-03T06:32:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:171691: waiting osd.0 seq 21474836542 2024-06-03T06:32:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:32:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836540 -lt 21474836542 2024-06-03T06:32:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:32:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:32:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:32:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836542 2024-06-03T06:32:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-347892350978 2024-06-03T06:32:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:32:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-347892350978 2024-06-03T06:32:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350978 2024-06-03T06:32:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 347892350978' 2024-06-03T06:32:49.620 INFO:tasks.workunit.client.0.smithi039.stderr:171691: waiting osd.1 seq 347892350978 2024-06-03T06:32:49.620 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:32:49.620 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350978 -lt 347892350978 2024-06-03T06:32:49.620 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:50.165 INFO:tasks.workunit.client.0.smithi039.stderr:1716tell osd.2 flush_pg_stats 2024-06-03T06:32:50.165 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579911 2024-06-03T06:32:50.165 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579911 2024-06-03T06:32:50.165 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-347892350979 2-317827579911' 2024-06-03T06:32:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:32:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:32:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252867 2024-06-03T06:32:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252867 2024-06-03T06:32:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-347892350979 2-317827579911 3-360777252867' 2024-06-03T06:32:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836543 2024-06-03T06:32:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:32:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836543 2024-06-03T06:32:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836543 2024-06-03T06:32:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-06-03T06:32:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:171689: waiting osd.0 seq 21474836543 2024-06-03T06:32:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:32:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836543 2024-06-03T06:32:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-347892350979 2024-06-03T06:32:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:32:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-347892350979 2024-06-03T06:32:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350979 2024-06-03T06:32:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 347892350979' 2024-06-03T06:32:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:171689: waiting osd.1 seq 347892350979 2024-06-03T06:32:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:32:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350978 -lt 347892350979 2024-06-03T06:32:50.168 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:32:50.168 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:32:50.168 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:32:50.168 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350979 -lt 347892350979 2024-06-03T06:32:50.168 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:50.908 INFO:tasks.workunit.client.0.smithi039.stderr:1791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579910 2024-06-03T06:32:50.909 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:50.909 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:32:50.909 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579910 2024-06-03T06:32:50.909 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:50.909 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579910 2024-06-03T06:32:50.909 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579910' 2024-06-03T06:32:50.909 INFO:tasks.workunit.client.0.smithi039.stderr:171691: waiting osd.2 seq 317827579910 2024-06-03T06:32:50.909 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:32:50.909 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579912 -lt 317827579910 2024-06-03T06:32:50.909 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:50.909 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-360777252866 2024-06-03T06:32:50.909 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:50.909 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:32:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-360777252866 2024-06-03T06:32:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-06-03T06:32:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 360777252866' 2024-06-03T06:32:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:171691: waiting osd.3 seq 360777252866 2024-06-03T06:32:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:32:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252866 2024-06-03T06:32:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:32:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:32:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:32:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:32:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:32:50.911 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:32:50.911 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:32:50.911 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:32:50.911 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:32:50.911 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:32:50.911 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:32:51.427 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.c1689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579911 2024-06-03T06:32:51.427 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:51.427 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:32:51.427 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579911 2024-06-03T06:32:51.427 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:51.427 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579911 2024-06-03T06:32:51.427 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579911' 2024-06-03T06:32:51.427 INFO:tasks.workunit.client.0.smithi039.stderr:171689: waiting osd.2 seq 317827579911 2024-06-03T06:32:51.427 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:32:51.427 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579912 -lt 317827579911 2024-06-03T06:32:51.427 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:32:51.428 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-360777252867 2024-06-03T06:32:51.428 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:32:51.428 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:32:51.428 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-360777252867 2024-06-03T06:32:51.428 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:32:51.428 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252867 2024-06-03T06:32:51.428 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 360777252867' 2024-06-03T06:32:51.428 INFO:tasks.workunit.client.0.smithi039.stderr:171689: waiting osd.3 seq 360777252867 2024-06-03T06:32:51.428 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:32:51.428 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252867 2024-06-03T06:32:51.428 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:32:51.428 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:51.428 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:51.428 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:32:51.429 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:32:51.429 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:32:51.429 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:32:51.429 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:32:51.429 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:32:51.429 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:32:51.429 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:32:51.429 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:32:51.429 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:32:52.047 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clonelient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:52.048 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:52.048 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-03T06:32:52.048 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-03T06:32:52.048 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:32:52.048 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-03T06:32:52.048 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:32:52.048 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:32:52.048 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:32:52.048 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:32:52.048 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:32:52.048 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:32:52.048 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:32:52.049 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:32:52.049 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:32:52.049 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:32:52.049 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:32:52.049 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:32:52.049 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:52.049 INFO:tasks.workunit.client.0.smithi039.stderr:171691: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:52.049 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:32:52.049 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:32:52.049 INFO:tasks.workunit.client.0.smithi039.stderr:171691: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:32:52.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:32:52.548 INFO:tasks.workunit.client.0.smithi039.stderr:.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:52.548 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:52.548 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-03T06:32:52.548 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-03T06:32:52.548 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:32:52.548 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-03T06:32:52.548 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:32:52.548 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:32:52.548 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:32:52.549 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:32:52.549 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:32:52.549 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:32:52.549 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:32:52.549 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:32:52.549 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:32:52.549 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:32:52.549 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:32:52.549 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:32:52.549 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:32:52.549 INFO:tasks.workunit.client.0.smithi039.stderr:171689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:32:52.549 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:32:52.550 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:32:52.550 INFO:tasks.workunit.client.0.smithi039.stderr:171689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:32:52.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:32:52.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:32:52.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 171688 2024-06-03T06:32:52.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:32:52.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:32:52.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:32:52.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-03T06:32:52.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:32:52.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-03T06:32:52.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:32:52.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:32:52.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:32:52.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:32:52.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-03T06:32:52.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-03T06:32:52.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:32:52.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:32:52.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:32:52.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:32:52.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:32:52.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:32:53.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:32:20.201148+0000 2024-06-03T06:32:53.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:32:53.422 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:32:53.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:32:20.201148+0000 2024-06-03T06:32:53.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:32:53.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:32:20.201148+0000 2024-06-03T06:32:53.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:32:53.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:32:53.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:32:53.434 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:32:53.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:32:53.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:32:53.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:32:53.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:32:53.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:32:20.201148+0000 '>' 2024-06-03T06:32:20.201148+0000 2024-06-03T06:32:53.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:32:54.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:32:54.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:32:54.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:32:54.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:32:54.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:32:54.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:32:54.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:32:55.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:32:20.201148+0000 '>' 2024-06-03T06:32:20.201148+0000 2024-06-03T06:32:55.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:32:56.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:32:56.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:32:56.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:32:56.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:32:56.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:32:56.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:32:56.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:32:56.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:32:20.201148+0000 '>' 2024-06-03T06:32:20.201148+0000 2024-06-03T06:32:56.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:32:57.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:32:57.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:32:57.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:32:57.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:32:57.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:32:57.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:32:57.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:32:57.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:32:20.201148+0000 '>' 2024-06-03T06:32:20.201148+0000 2024-06-03T06:32:57.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:32:58.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:32:58.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:32:58.620 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:32:58.620 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:32:58.620 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:32:58.620 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:32:58.620 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:32:58.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:32:53.487082+0000 '>' 2024-06-03T06:32:20.201148+0000 2024-06-03T06:32:58.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:32:58.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-03T06:32:58.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:32:58.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:32:58.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:32:58.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 174636"' 2024-06-03T06:32:58.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 174636' 2024-06-03T06:32:58.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:32:58.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:32:58.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:32:58.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 174638"' 2024-06-03T06:32:58.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 174638' 2024-06-03T06:32:58.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-03T06:32:58.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 174636 174638' 2024-06-03T06:32:58.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:32:58.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:32:58.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:32:58.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 174636 2024-06-03T06:32:58.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/174639: /' 2024-06-03T06:32:58.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:32:58.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/174641: /' 2024-06-03T06:33:01.370 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:33:01.370 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:33:01.370 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:33:01.370 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:33:01.370 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:33:01.370 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:33:01.370 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:33:01.370 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:33:01.370 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:33:01.370 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:33:01.370 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:33:01.371 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:33:01.371 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:33:01.371 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:33:01.371 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:33:01.371 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:33:01.371 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:33:01.371 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:33:01.371 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:33:01.371 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:33:01.371 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:33:01.371 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:33:01.373 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-03T06:33:01.373 INFO:tasks.workunit.client.0.smithi039.stderr:174639: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-03T06:33:01.373 INFO:tasks.workunit.client.0.smithi039.stderr:174639: _ 2024-06-03T06:33:01.373 INFO:tasks.workunit.client.0.smithi039.stderr:174639: hinfo_key 2024-06-03T06:33:01.373 INFO:tasks.workunit.client.0.smithi039.stderr:174639: snapset 2024-06-03T06:33:01.374 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:33:01.374 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:33:01.374 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:33:01.374 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:33:01.374 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:33:01.374 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:33:01.374 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:33:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:33:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:33:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:33:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:33:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:33:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:33:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:33:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:33:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:33:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:33:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:33:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:174639: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:33:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:174639: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:33:01.376 INFO:tasks.workunit.client.0.smithi039.stderr:174639: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:33:01.376 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:33:01.376 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:33:01.376 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:33:01.376 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:33:01.376 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:33:01.376 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:33:01.376 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:33:01.376 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:33:01.376 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:33:01.376 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:33:01.376 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:33:01.376 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:33:01.376 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:33:01.377 INFO:tasks.workunit.client.0.smithi039.stderr:174639: start osd.3 2024-06-03T06:33:01.752 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:33:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:33:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:33:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:33:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:33:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:33:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:33:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:33:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:33:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:33:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:33:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:33:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:33:01.754 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:33:01.754 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:33:01.754 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:33:01.754 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:33:01.754 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:33:01.754 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:33:01.754 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:33:01.754 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:33:01.754 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:33:01.754 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-03T06:33:01.754 INFO:tasks.workunit.client.0.smithi039.stderr:174641: _ 2024-06-03T06:33:01.754 INFO:tasks.workunit.client.0.smithi039.stderr:174641: hinfo_key 2024-06-03T06:33:01.754 INFO:tasks.workunit.client.0.smithi039.stderr:174641: snapset 2024-06-03T06:33:01.755 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:33:01.755 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:33:01.755 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:33:01.755 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:33:01.755 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:33:01.755 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:33:01.755 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:33:01.755 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:33:01.755 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:33:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:33:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:33:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:33:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:33:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:33:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:33:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:33:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:33:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:33:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:174641: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:33:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:174641: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:33:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:174641: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:33:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:33:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:33:01.757 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:33:01.757 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:33:01.757 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:33:01.757 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:33:01.757 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:33:01.757 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:33:01.757 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:33:01.757 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:33:01.757 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:33:01.757 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:33:01.757 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:33:01.757 INFO:tasks.workunit.client.0.smithi039.stderr:174641: start osd.1 2024-06-03T06:33:05.614 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:33:05.614 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:33:05.614 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:33:05.614 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:33:05.614 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:33:05.614 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:33:05.614 INFO:tasks.workunit.client.0.smithi039.stderr:174639: 2024-06-03T06:33:01.403+0000 7f03237da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:174639: 2024-06-03T06:33:01.406+0000 7f03237da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:174639: 2024-06-03T06:33:01.407+0000 7f03237da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:174639: 0 2024-06-03T06:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:174639: 2024-06-03T06:33:01.921+0000 7f03237da640 -1 Falling back to public interface 2024-06-03T06:33:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:33:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:174639: 1 2024-06-03T06:33:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:33:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:33:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:174639: 2 2024-06-03T06:33:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:33:05.617 INFO:tasks.workunit.client.0.smithi039.stderr:174639: 2024-06-03T06:33:04.450+0000 7f03237da640 -1 osd.3 86 log_to_monitors true 2024-06-03T06:33:05.617 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:05.617 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:05.617 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:05.617 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:33:05.617 INFO:tasks.workunit.client.0.smithi039.stderr:174639: 3 2024-06-03T06:33:05.617 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-06-03T06:33:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:33:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:33:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:33:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:33:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:33:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: 2024-06-03T06:33:01.783+0000 7f851e617640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: 2024-06-03T06:33:01.786+0000 7f851e617640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: 2024-06-03T06:33:01.787+0000 7f851e617640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:33:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:33:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:33:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:33:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:33:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:33:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:174641: 0 2024-06-03T06:33:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:33:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:174641: 2024-06-03T06:33:02.308+0000 7f851e617640 -1 Falling back to public interface 2024-06-03T06:33:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:33:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:174641: 1 2024-06-03T06:33:06.351 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:06.351 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:33:06.351 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:06.351 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:06.351 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:06.351 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:33:06.351 INFO:tasks.workunit.client.0.smithi039.stderr:174641: 2 2024-06-03T06:33:06.351 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:06.351 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:33:06.351 INFO:tasks.workunit.client.0.smithi039.stderr:174641: 2024-06-03T06:33:04.976+0000 7f851e617640 -1 osd.1 86 log_to_monitors true 2024-06-03T06:33:06.351 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:06.351 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:06.351 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:06.352 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:33:06.352 INFO:tasks.workunit.client.0.smithi039.stderr:174641: 3 2024-06-03T06:33:06.352 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:06.352 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:33:06.910 INFO:tasks.workunit.client.0.smithi039.stderr:174641: osd.1 up in weight 1 up_from 94 up_thru 81 down_as.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:33:06.910 INFO:tasks.workunit.client.0.smithi039.stderr:174639: osd.3 up in weight 1 up_from 91 up_thru 91 down_at 87 last_clean_interval [84,86) [v2:127.0.0.1:6818/1379649895,v1:127.0.0.1:6819/1379649895] [v2:127.0.0.1:6820/1379649895,v1:127.0.0.1:6821/1379649895] exists,up c13f28c9-c6e3-4efc-acdf-7e5d713b78ec 2024-06-03T06:33:06.910 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:33:06.910 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:33:06.910 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:33:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:33:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:33:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:33:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:33:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:33:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:174639: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:33:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:174639: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:33:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:33:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:33:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:33:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:33:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:33:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:33:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:33:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:33:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:33:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:33:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:174639: 1 2024-06-03T06:33:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:174639: 2 2024-06-03T06:33:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:174639: 3' 2024-06-03T06:33:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:33:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:33:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:33:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836548 2024-06-03T06:33:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836548 2024-06-03T06:33:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548' 2024-06-03T06:33:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:33:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:33:06.913 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925826 2024-06-03T06:33:06.913 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925826 2024-06-03T06:33:06.913 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-403726925826' 2024-06-03T06:33:07.222 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd it 87 last_clean_interval [81,86) [v2:127.0.0.1:6826/4049176407,v1:127.0.0.1:6827/4049176407] [v2:127.0.0.1:6828/4049176407,v1:127.0.0.1:6829/4049176407] exists,up a68c99d8-3111-4a07-aff9-9e8216641514 2024-06-03T06:33:07.222 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:33:07.222 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:33:07.223 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:33:07.223 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:33:07.223 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:33:07.223 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:33:07.223 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:33:07.223 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:33:07.223 INFO:tasks.workunit.client.0.smithi039.stderr:174641: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:33:07.223 INFO:tasks.workunit.client.0.smithi039.stderr:174641: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:33:07.223 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:33:07.223 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:33:07.223 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:33:07.223 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:33:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:33:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:33:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:33:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:33:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:33:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:33:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:174641: 1 2024-06-03T06:33:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:174641: 2 2024-06-03T06:33:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:174641: 3' 2024-06-03T06:33:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:33:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:33:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:33:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836549 2024-06-03T06:33:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836549 2024-06-03T06:33:07.225 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549' 2024-06-03T06:33:07.225 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:33:07.225 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:33:07.225 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925827 2024-06-03T06:33:07.225 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925827 2024-06-03T06:33:07.225 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-403726925827' 2024-06-03T06:33:07.225 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:33:08.346 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:33:08.346 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579917 2024-06-03T06:33:08.346 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579917 2024-06-03T06:33:08.346 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-403726925827 2-317827579917' 2024-06-03T06:33:08.347 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:33:08.347 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:33:08.347 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023939 2024-06-03T06:33:08.347 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023939 2024-06-03T06:33:08.347 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-403726925827 2-317827579917 3-390842023939' 2024-06-03T06:33:08.347 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:33:08.347 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836549 2024-06-03T06:33:08.347 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:33:08.347 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:33:08.347 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836549 2024-06-03T06:33:08.347 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:33:08.347 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836549 2024-06-03T06:33:08.348 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836549' 2024-06-03T06:33:08.348 INFO:tasks.workunit.client.0.smithi039.stderr:174641: waiting osd.0 seq 21474836549 2024-06-03T06:33:08.348 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:33:08.348 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836549 2024-06-03T06:33:08.348 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:33:08.348 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-403726925827 2024-06-03T06:33:08.348 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:33:08.348 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:33:08.348 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-403726925827 2024-06-03T06:33:08.348 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:33:08.348 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925827 2024-06-03T06:33:08.348 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 403726925827' 2024-06-03T06:33:08.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: waiting osd.1 seq 403726925827 2024-06-03T06:33:08.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:33:08.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925827 2024-06-03T06:33:08.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:33:08.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579917 2024-06-03T06:33:08.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:33:08.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:33:08.349 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579917 2024-06-03T06:33:09.430 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/un $ids 2024-06-03T06:33:09.431 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:33:09.431 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579916 2024-06-03T06:33:09.431 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579916 2024-06-03T06:33:09.431 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-403726925826 2-317827579916' 2024-06-03T06:33:09.431 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:33:09.431 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:33:09.431 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023938 2024-06-03T06:33:09.431 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023938 2024-06-03T06:33:09.431 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-403726925826 2-317827579916 3-390842023938' 2024-06-03T06:33:09.431 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:33:09.431 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836548 2024-06-03T06:33:09.431 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:33:09.431 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:33:09.431 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836548 2024-06-03T06:33:09.432 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:33:09.432 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836548 2024-06-03T06:33:09.432 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-06-03T06:33:09.432 INFO:tasks.workunit.client.0.smithi039.stderr:174639: waiting osd.0 seq 21474836548 2024-06-03T06:33:09.432 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:33:09.432 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836546 -lt 21474836548 2024-06-03T06:33:09.432 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:33:09.432 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:33:09.432 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:33:09.432 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836548 2024-06-03T06:33:09.432 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:33:09.432 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-403726925826 2024-06-03T06:33:09.432 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:33:09.432 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:33:09.433 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-403726925826 2024-06-03T06:33:09.433 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:33:09.433 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925826 2024-06-03T06:33:09.433 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 403726925826' 2024-06-03T06:33:09.433 INFO:tasks.workunit.client.0.smithi039.stderr:174639: waiting osd.1 seq 403726925826 2024-06-03T06:33:09.433 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:33:09.433 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925826 2024-06-03T06:33:10.681 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:33:10.681 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579916 2024-06-03T06:33:10.681 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:33:10.681 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:33:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579916 2024-06-03T06:33:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:33:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579916 2024-06-03T06:33:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579916' 2024-06-03T06:33:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:174639: waiting osd.2 seq 317827579916 2024-06-03T06:33:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:33:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579917 -lt 317827579916 2024-06-03T06:33:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:33:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-390842023938 2024-06-03T06:33:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:33:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:33:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-390842023938 2024-06-03T06:33:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:33:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023938 2024-06-03T06:33:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 390842023938' 2024-06-03T06:33:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:174639: waiting osd.3 seq 390842023938 2024-06-03T06:33:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:33:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023939 -lt 390842023938 2024-06-03T06:33:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:33:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:33:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:33:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:33:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:33:10.683 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:33:10.684 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:33:10.684 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:33:10.684 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:33:10.684 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:33:10.684 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:33:10.684 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-03T06:33:10.978 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.cbuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:33:10.978 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579917 2024-06-03T06:33:10.979 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579917' 2024-06-03T06:33:10.979 INFO:tasks.workunit.client.0.smithi039.stderr:174641: waiting osd.2 seq 317827579917 2024-06-03T06:33:10.979 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:33:10.979 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579917 -lt 317827579917 2024-06-03T06:33:10.979 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:33:10.979 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-390842023939 2024-06-03T06:33:10.979 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:33:10.979 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:33:10.979 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-390842023939 2024-06-03T06:33:10.979 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:33:10.979 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023939 2024-06-03T06:33:10.979 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 390842023939' 2024-06-03T06:33:10.979 INFO:tasks.workunit.client.0.smithi039.stderr:174641: waiting osd.3 seq 390842023939 2024-06-03T06:33:10.980 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:33:10.980 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023938 -lt 390842023939 2024-06-03T06:33:10.980 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:33:10.980 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:33:10.980 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:33:10.980 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023939 -lt 390842023939 2024-06-03T06:33:10.980 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:33:10.980 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:33:10.980 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:33:10.980 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:33:10.980 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:33:10.980 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:33:10.980 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:33:10.981 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:33:10.981 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:33:10.981 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:33:10.981 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:33:10.981 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-03T06:33:10.981 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:33:12.136 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:33:12.137 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:33:12.137 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-03T06:33:12.137 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-03T06:33:12.137 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:33:12.137 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-03T06:33:12.137 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:33:12.137 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:33:12.137 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:33:12.137 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:33:12.137 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:33:12.137 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:33:12.138 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:33:12.138 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:33:12.138 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:33:12.138 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:33:12.138 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:33:12.138 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:33:12.138 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:33:12.138 INFO:tasks.workunit.client.0.smithi039.stderr:174641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:33:12.138 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:33:12.138 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:33:12.138 INFO:tasks.workunit.client.0.smithi039.stderr:174641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:33:12.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:33:12.266 INFO:tasks.workunit.client.0.smithi039.stderr:lient.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:33:12.266 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:33:12.266 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:33:12.266 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-03T06:33:12.266 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-03T06:33:12.266 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:33:12.266 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-03T06:33:12.266 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:33:12.266 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:33:12.267 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:33:12.267 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:33:12.267 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:33:12.267 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:33:12.267 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:33:12.267 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:33:12.267 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:33:12.267 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:33:12.267 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-03T06:33:12.267 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:33:12.267 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:33:12.267 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:33:12.268 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-03T06:33:12.268 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-06-03T06:33:12.268 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:33:12.268 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:33:12.268 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:33:12.268 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:33:12.268 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:33:12.268 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:33:12.268 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:33:12.268 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-06-03T06:33:13.016 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-06-03T06:33:13.016 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-06-03T06:33:13.016 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:33:13.016 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:33:13.016 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:33:13.016 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:33:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:33:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:33:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:33:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:33:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:33:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:33:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:33:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:33:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:33:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:174639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:33:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:33:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:33:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:174639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:33:13.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:33:13.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:33:13.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 174638 2024-06-03T06:33:13.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:33:13.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:33:13.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:33:13.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-03T06:33:13.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:33:13.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:33:13.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:33:13.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T06:33:13.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:33:13.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:33:13.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:33:13.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:33:13.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:33:13.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:33:13.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:33:13.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:33:13.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:33:13.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:33:13.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:33:13.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:33:13.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:33:13.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:33:13.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:33:13.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:33:13.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:33:13.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:33:13.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:33:13.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:33:13.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:33:13.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:33:13.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:33:13.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:33:13.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:33:13.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:33:13.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:33:13.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:33:13.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T06:33:13.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T06:33:13.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T06:33:13.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T06:33:13.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:33:13.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:33:13.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:33:13.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:33:13.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:33:13.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:33:13.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:33:13.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:33:13.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:33:13.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:33:13.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:33:13.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:33:13.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:33:13.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:33:13.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:33:13.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:33:13.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:33:13.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:33:13.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:33:13.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:33:13.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:33:13.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:33:13.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:33:13.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:33:13.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:33:13.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:33:13.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:33:13.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:33:13.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T06:33:13.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:33:13.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:33:13.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:33:13.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T06:33:13.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:33:13.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T06:33:13.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:33:13.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T06:33:13.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-06-03T06:33:13.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-06-03T06:33:13.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-06-03T06:33:13.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-06-03T06:33:13.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-06-03T06:33:13.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-06-03T06:33:13.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-06-03T06:33:13.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T06:33:13.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T06:33:13.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T06:33:13.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T06:33:13.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T06:33:13.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T06:33:13.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T06:33:13.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:33:13.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:33:13.579 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:33:13.579 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:33:13.579 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:33:13.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:33:13.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T06:33:13.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T06:33:13.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T06:33:13.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:33:13.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:33:13.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T06:33:13.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T06:33:13.784 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:33:13.784 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:33:13.785 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:33:13.785 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:33:13.785 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:33:13.785 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:33:13.785 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:33:13.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:33:13.786 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T06:33:13.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T06:33:13.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:33:13.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:33:13.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T06:33:13.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T06:33:13.870 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:33:13.870 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:33:13.871 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:33:13.871 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:33:13.871 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:33:13.871 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:33:13.871 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:33:13.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:33:13.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T06:33:13.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-06-03T06:33:13.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T06:33:13.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T06:33:13.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T06:33:13.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T06:33:13.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T06:33:13.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T06:33:14.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T06:33:14.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:33:14.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:33:14.775 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:33:14.775 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:33:14.775 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:33:14.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:33:14.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:33:14.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:33:14.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-06-03T06:33:14.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-03T06:33:14.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-06-03T06:33:14.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:33:14.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:33:14.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T06:33:14.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:33:14.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:33:14.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:33:14.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:33:14.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:33:14.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:33:14.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:33:14.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:33:14.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:33:14.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:33:14.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:33:14.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:33:14.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:33:14.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:33:14.810 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:33:14.810 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:33:14.810 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:33:14.810 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:33:14.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:33:14.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:33:14.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:33:14.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:33:14.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:33:14.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:33:14.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:33:14.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:33:14.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:33:14.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:33:14.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:33:14.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:33:14.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:33:14.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:33:14.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1a2b1190-8afb-4060-bea8-4c937d61d293 2024-06-03T06:33:14.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1a2b1190-8afb-4060-bea8-4c937d61d293' 2024-06-03T06:33:14.812 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 1a2b1190-8afb-4060-bea8-4c937d61d293 2024-06-03T06:33:14.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:33:14.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCqY11mJZ1QMRAAM7W2XT/6hbYORxr1rdYsQw== 2024-06-03T06:33:14.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCqY11mJZ1QMRAAM7W2XT/6hbYORxr1rdYsQw=="}' 2024-06-03T06:33:14.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1a2b1190-8afb-4060-bea8-4c937d61d293 -i td/osd-scrub-repair/0/new.json 2024-06-03T06:33:15.088 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:33:15.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T06:33:15.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCqY11mJZ1QMRAAM7W2XT/6hbYORxr1rdYsQw== --osd-uuid 1a2b1190-8afb-4060-bea8-4c937d61d293 2024-06-03T06:33:15.129 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:15.128+0000 7f1257e26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:15.139 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:15.138+0000 7f1257e26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:15.139+0000 7f1257e26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:15.140+0000 7f1257e26640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:33:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:15.140+0000 7f1257e26640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:33:15.141 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:15.140+0000 7f1257e26640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:33:15.231 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:15.230+0000 7f1257e26640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T06:33:20.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T06:33:20.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:33:20.776 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:33:20.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T06:33:20.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:33:21.156 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:33:21.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T06:33:21.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:33:21.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:33:21.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:33:21.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:33:21.184 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:21.183+0000 7f5f4be11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:21.186 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:21.185+0000 7f5f4be11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:21.187 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:21.186+0000 7f5f4be11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:21.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T06:33:21.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:33:21.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:33:21.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:33:21.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:33:21.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:21.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:33:21.446 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:33:21.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:21.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:21.701 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:21.701+0000 7f5f4be11640 -1 Falling back to public interface 2024-06-03T06:33:21.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:22.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:22.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:22.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:33:22.736 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:33:22.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:22.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:23.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:24.014 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:24.013+0000 7f5f4be11640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:33:24.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:24.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:24.042 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:33:24.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:33:24.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:24.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:24.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:25.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:25.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:25.440 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:33:25.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:33:25.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:25.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:25.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:26.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:26.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:26.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:33:26.762 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:33:26.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:26.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:27.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:28.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:28.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:28.062 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:33:28.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:33:28.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:28.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:28.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:29.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:29.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:29.367 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:33:29.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:33:29.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:29.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:29.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:30.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:30.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:30.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:33:30.746 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:33:30.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:30.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:31.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:32.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:32.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:32.046 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:33:32.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:33:32.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:32.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:32.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:33.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:33.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:33.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:33:33.346 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:33:33.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:33.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:33.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:34.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:34.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:34.772 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:33:34.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:33:34.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:34.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:35.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:36.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:36.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:36.074 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:33:36.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:33:36.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:36.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:36.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:37.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:37.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:37.745 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:33:37.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:33:37.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:37.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:38.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:39.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:39.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:39.278 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:33:39.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:33:39.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:39.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:40.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:41.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:41.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:41.031 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:33:41.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:33:41.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:41.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:41.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:42.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:42.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:42.466 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:33:42.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:33:42.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:42.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:42.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:43.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:43.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:43.965 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:33:43.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:33:43.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:43.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:44.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:45.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:45.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:45.278 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:33:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:33:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:45.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:46.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:46.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:46.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:33:47.000 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:33:47.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:47.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:33:47.298 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3137609328,v1:127.0.0.1:6803/3137609328] [v2:127.0.0.1:6804/3137609328,v1:127.0.0.1:6805/3137609328] exists,up 1a2b1190-8afb-4060-bea8-4c937d61d293 2024-06-03T06:33:47.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:33:47.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:33:47.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:33:47.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-03T06:33:47.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-06-03T06:33:47.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:33:47.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:33:47.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T06:33:47.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:33:47.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:33:47.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:33:47.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:33:47.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:33:47.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:33:47.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:33:47.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:33:47.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:33:47.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:33:47.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:33:47.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:33:47.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:33:47.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:33:47.304 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:33:47.305 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:33:47.305 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:33:47.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:33:47.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:33:47.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:33:47.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:33:47.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:33:47.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:33:47.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:33:47.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:33:47.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:33:47.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:33:47.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:33:47.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:33:47.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:33:47.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:33:47.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:33:47.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c1bf0b94-99bf-4fb0-823c-5047aa4d2dd5 2024-06-03T06:33:47.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c1bf0b94-99bf-4fb0-823c-5047aa4d2dd5' 2024-06-03T06:33:47.306 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 c1bf0b94-99bf-4fb0-823c-5047aa4d2dd5 2024-06-03T06:33:47.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:33:47.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLY11mBNoZExAAXk9WpgygGrGuevme4MuMgA== 2024-06-03T06:33:47.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLY11mBNoZExAAXk9WpgygGrGuevme4MuMgA=="}' 2024-06-03T06:33:47.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c1bf0b94-99bf-4fb0-823c-5047aa4d2dd5 -i td/osd-scrub-repair/1/new.json 2024-06-03T06:33:47.666 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:33:47.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T06:33:47.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLY11mBNoZExAAXk9WpgygGrGuevme4MuMgA== --osd-uuid c1bf0b94-99bf-4fb0-823c-5047aa4d2dd5 2024-06-03T06:33:47.707 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:47.705+0000 7fa0fe108640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:47.750 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:47.749+0000 7fa0fe108640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:47.751 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:47.750+0000 7fa0fe108640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:47.752 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:47.750+0000 7fa0fe108640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:33:47.752 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:47.750+0000 7fa0fe108640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:33:47.752 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:47.750+0000 7fa0fe108640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:33:47.825 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:47.824+0000 7fa0fe108640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T06:33:53.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T06:33:53.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:33:53.674 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:33:53.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T06:33:53.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:33:54.050 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:33:54.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T06:33:54.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:33:54.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:33:54.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:33:54.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:33:54.077 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:54.077+0000 7ff436625640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:54.079 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:54.079+0000 7ff436625640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:54.080 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:54.080+0000 7ff436625640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:33:54.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T06:33:54.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:33:54.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:33:54.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:33:54.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:33:54.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:54.339 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:33:54.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:33:54.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:54.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:33:54.604 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:54.603+0000 7ff436625640 -1 Falling back to public interface 2024-06-03T06:33:54.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:55.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:55.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:55.630 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:33:55.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:33:55.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:55.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:33:55.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:56.957 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:33:56.957+0000 7ff436625640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:33:56.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:56.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:56.971 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:33:56.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:33:56.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:56.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:33:57.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:58.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:58.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:58.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:33:58.375 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:33:58.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:58.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:33:58.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:33:59.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:33:59.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:33:59.714 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:33:59.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:33:59.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:33:59.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:34:00.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:01.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:01.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:01.080 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:34:01.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:34:01.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:01.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:34:01.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:02.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:02.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:02.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:34:02.380 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:34:02.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:02.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:34:02.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:03.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:03.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:03.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:34:03.687 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:34:03.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:03.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:34:03.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:04.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:04.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:04.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:34:04.989 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:34:04.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:04.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:34:05.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:06.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:06.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:06.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:34:06.289 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:34:06.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:06.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:34:06.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:07.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:07.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:07.588 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:34:07.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:34:07.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:07.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:34:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:08.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:08.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:08.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:34:08.936 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:34:08.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:08.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:34:09.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:10.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:10.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:10.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:34:10.613 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:34:10.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:10.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:34:11.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:12.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:12.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:12.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:34:12.128 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:34:12.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:12.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:34:12.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:13.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:13.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:13.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:34:13.757 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:34:13.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:13.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:34:14.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:15.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:15.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:15.315 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:34:15.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:34:15.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:15.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:34:16.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:17.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:17.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:17.027 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:34:17.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:34:17.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:17.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:34:17.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:18.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:18.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:18.745 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:34:18.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:34:18.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:18.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:34:19.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:20.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:20.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:20.365 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:34:20.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:34:20.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:20.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:34:20.712 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1607284046,v1:127.0.0.1:6811/1607284046] [v2:127.0.0.1:6812/1607284046,v1:127.0.0.1:6813/1607284046] exists,up c1bf0b94-99bf-4fb0-823c-5047aa4d2dd5 2024-06-03T06:34:20.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:34:20.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:34:20.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:34:20.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-03T06:34:20.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-06-03T06:34:20.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:34:20.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:34:20.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T06:34:20.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:34:20.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:34:20.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:34:20.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:34:20.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:34:20.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:34:20.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:34:20.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:34:20.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:34:20.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:34:20.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:34:20.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:34:20.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:34:20.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:34:20.718 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:34:20.718 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:34:20.719 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:34:20.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:34:20.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:34:20.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:34:20.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:34:20.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:34:20.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:34:20.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:34:20.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:34:20.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:34:20.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:34:20.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:34:20.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:34:20.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:34:20.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:34:20.720 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:34:20.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5cd6cf24-6678-4c6e-9721-cd271fc1f0af 2024-06-03T06:34:20.720 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 5cd6cf24-6678-4c6e-9721-cd271fc1f0af 2024-06-03T06:34:20.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5cd6cf24-6678-4c6e-9721-cd271fc1f0af' 2024-06-03T06:34:20.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:34:20.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsY11mE97IKxAAsaBU5EnOmP2hJJ1ITJmusw== 2024-06-03T06:34:20.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsY11mE97IKxAAsaBU5EnOmP2hJJ1ITJmusw=="}' 2024-06-03T06:34:20.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5cd6cf24-6678-4c6e-9721-cd271fc1f0af -i td/osd-scrub-repair/2/new.json 2024-06-03T06:34:21.075 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:34:21.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-03T06:34:21.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsY11mE97IKxAAsaBU5EnOmP2hJJ1ITJmusw== --osd-uuid 5cd6cf24-6678-4c6e-9721-cd271fc1f0af 2024-06-03T06:34:21.116 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:21.116+0000 7f342fc43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:34:21.159 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:21.159+0000 7f342fc43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:34:21.160 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:21.160+0000 7f342fc43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:34:21.161 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:21.160+0000 7f342fc43640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:34:21.161 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:21.160+0000 7f342fc43640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:34:21.161 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:21.160+0000 7f342fc43640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:34:21.226 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:21.225+0000 7f342fc43640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T06:34:27.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-03T06:34:27.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:34:27.316 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T06:34:27.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T06:34:27.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:34:27.690 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:34:27.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T06:34:27.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:34:27.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:34:27.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:34:27.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:34:27.717 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:27.717+0000 7f5f09e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:34:27.719 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:27.719+0000 7f5f09e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:34:27.720 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:27.720+0000 7f5f09e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:34:27.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T06:34:27.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:34:27.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:34:27.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:34:27.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:34:27.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:27.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:34:27.989 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:34:27.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:27.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:28.239 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:28.239+0000 7f5f09e11640 -1 Falling back to public interface 2024-06-03T06:34:28.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:29.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:29.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:29.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:34:29.279 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:34:29.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:29.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:29.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:30.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:30.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:30.578 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:34:30.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:34:30.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:30.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:30.602 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:30.602+0000 7f5f09e11640 -1 osd.2 0 log_to_monitors true 2024-06-03T06:34:30.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:31.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:31.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:31.911 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:34:31.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:34:31.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:31.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:32.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:33.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:33.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:33.326 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:34:33.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:34:33.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:33.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:33.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:34.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:34.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:34.633 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:34:34.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:34:34.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:34.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:34.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:35.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:35.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:35.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:34:35.935 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:34:35.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:35.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:36.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:37.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:37.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:37.336 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:34:37.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:34:37.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:37.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:37.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:38.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:38.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:38.656 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:34:38.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:34:38.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:38.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:38.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:39.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:39.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:39.957 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:34:39.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:34:39.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:39.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:40.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:41.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:41.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:41.372 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:34:41.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:34:41.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:41.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:41.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:42.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:42.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:42.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:34:42.674 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:34:42.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:42.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:43.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:44.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:44.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:44.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:34:44.258 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:34:44.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:44.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:44.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:45.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:45.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:45.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:34:45.907 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:34:45.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:45.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:46.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:47.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:47.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:47.668 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:34:47.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:34:47.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:47.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:48.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:49.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:49.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:49.268 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:34:49.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:34:49.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:49.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:49.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:50.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:50.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:50.653 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:34:50.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:34:50.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:50.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:50.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:51.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:51.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:51.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:34:51.955 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:34:51.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:51.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:52.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:53.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:53.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:53.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:34:53.467 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:34:53.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:53.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:53.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:34:54.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:34:54.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:34:54.769 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:34:54.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:34:54.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:34:54.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:34:55.088 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2936077789,v1:127.0.0.1:6819/2936077789] [v2:127.0.0.1:6820/2936077789,v1:127.0.0.1:6821/2936077789] exists,up 5cd6cf24-6678-4c6e-9721-cd271fc1f0af 2024-06-03T06:34:55.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:34:55.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:34:55.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:34:55.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-03T06:34:55.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-06-03T06:34:55.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:34:55.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:34:55.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-03T06:34:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:34:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:34:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:34:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:34:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:34:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:34:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:34:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:34:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:34:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:34:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:34:55.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:34:55.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:34:55.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:34:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:34:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:34:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:34:55.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:34:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:34:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:34:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:34:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:34:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:34:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:34:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:34:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:34:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:34:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:34:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:34:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:34:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:34:55.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:34:55.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6f805989-769d-4078-8009-2c052ac9acc1 2024-06-03T06:34:55.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 6f805989-769d-4078-8009-2c052ac9acc1' 2024-06-03T06:34:55.099 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 6f805989-769d-4078-8009-2c052ac9acc1 2024-06-03T06:34:55.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:34:55.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPZF1mrpSiBhAAdCfw7ZLBhM6ssXLqSaF9Aw== 2024-06-03T06:34:55.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPZF1mrpSiBhAAdCfw7ZLBhM6ssXLqSaF9Aw=="}' 2024-06-03T06:34:55.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6f805989-769d-4078-8009-2c052ac9acc1 -i td/osd-scrub-repair/3/new.json 2024-06-03T06:34:55.495 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:34:55.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-06-03T06:34:55.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPZF1mrpSiBhAAdCfw7ZLBhM6ssXLqSaF9Aw== --osd-uuid 6f805989-769d-4078-8009-2c052ac9acc1 2024-06-03T06:34:55.536 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:55.536+0000 7fef2640a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:34:55.605 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:55.604+0000 7fef2640a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:34:55.606 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:55.605+0000 7fef2640a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:34:55.606 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:55.606+0000 7fef2640a640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-03T06:34:55.607 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:55.606+0000 7fef2640a640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-03T06:34:55.607 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:55.606+0000 7fef2640a640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-03T06:34:55.696 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:34:55.696+0000 7fef2640a640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-06-03T06:35:01.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-06-03T06:35:01.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:35:01.445 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-06-03T06:35:01.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-03T06:35:01.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:35:01.826 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-06-03T06:35:01.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-03T06:35:01.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:35:01.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:35:01.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:35:01.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:35:01.853 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:35:01.852+0000 7fc05d80e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:35:01.855 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:35:01.855+0000 7fc05d80e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:35:01.856 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:35:01.856+0000 7fc05d80e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:35:02.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-03T06:35:02.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:35:02.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:35:02.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:35:02.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:35:02.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:02.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:35:02.118 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:35:02.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:02.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:02.374 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:35:02.374+0000 7fc05d80e640 -1 Falling back to public interface 2024-06-03T06:35:02.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:03.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:03.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:03.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:35:03.406 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:35:03.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:03.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:03.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:04.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:04.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:04.706 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:35:04.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:35:04.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:04.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:04.733 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:35:04.733+0000 7fc05d80e640 -1 osd.3 0 log_to_monitors true 2024-06-03T06:35:05.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:06.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:06.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:06.059 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:35:06.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:35:06.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:06.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:06.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:07.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:07.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:07.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:35:07.367 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:35:07.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:07.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:07.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:08.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:08.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:08.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:35:08.748 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:35:08.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:08.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:09.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:10.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:10.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:10.050 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:35:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:35:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:10.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:11.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:11.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:11.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:35:11.393 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:35:11.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:11.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:11.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:12.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:12.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:12.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:35:12.733 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:35:12.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:12.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:13.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:14.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:14.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:14.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:35:14.064 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:35:14.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:14.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:14.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:15.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:15.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:15.366 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:35:15.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:35:15.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:15.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:15.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:16.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:16.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:16.726 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:35:16.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:35:16.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:16.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:18.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:18.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:18.259 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:35:18.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:35:18.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:18.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:18.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:19.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:19.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:19.797 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:35:19.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:35:19.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:19.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:20.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:21.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:21.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:21.417 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:35:21.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:35:21.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:21.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:21.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:22.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:22.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:22.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:35:22.867 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:35:22.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:22.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:23.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:24.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:24.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:24.352 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:35:24.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:35:24.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:24.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:24.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:25.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:25.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:25.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:35:25.914 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:35:25.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:25.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:26.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:27.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:27.436 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:35:27.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:27.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:35:27.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:27.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:27.753 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2541525872,v1:127.0.0.1:6827/2541525872] [v2:127.0.0.1:6828/2541525872,v1:127.0.0.1:6829/2541525872] exists,up 6f805989-769d-4078-8009-2c052ac9acc1 2024-06-03T06:35:27.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:35:27.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:35:27.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:35:27.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-06-03T06:35:27.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T06:35:28.033 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-06-03T06:35:28.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T06:35:28.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T06:35:28.530 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T06:35:28.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:35:29.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T06:35:31.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-06-03T06:35:31.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:35:31.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:35:31.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:35:31.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:35:31.874 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:35:31.874 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:35:31.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:35:31.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:35:31.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:35:31.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:35:31.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:35:31.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:35:31.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:35:31.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:35:31.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:35:32.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:35:32.271 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:35:32.271 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:35:32.271 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-06-03T06:35:32.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:35:32.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:35:32.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:35:32.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836511 2024-06-03T06:35:32.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836511 2024-06-03T06:35:32.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511' 2024-06-03T06:35:32.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:35:32.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:35:32.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-06-03T06:35:32.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-06-03T06:35:32.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672981' 2024-06-03T06:35:32.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:35:32.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:35:32.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-06-03T06:35:32.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-06-03T06:35:32.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672981 2-64424509454' 2024-06-03T06:35:32.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:35:32.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:35:32.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345926 2024-06-03T06:35:32.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345926 2024-06-03T06:35:32.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672981 2-64424509454 3-85899345926' 2024-06-03T06:35:32.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:35:32.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836511 2024-06-03T06:35:32.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:35:32.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:35:32.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836511 2024-06-03T06:35:32.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:35:32.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836511 2024-06-03T06:35:32.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-06-03T06:35:32.705 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836511 2024-06-03T06:35:32.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:35:33.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836511 2024-06-03T06:35:33.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:35:34.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:35:34.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:35:34.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836511 2024-06-03T06:35:34.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:35:34.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-06-03T06:35:34.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:35:34.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:35:34.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-06-03T06:35:34.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:35:34.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-06-03T06:35:34.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-06-03T06:35:34.320 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672981 2024-06-03T06:35:34.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:35:34.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-06-03T06:35:34.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:35:34.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-06-03T06:35:34.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:35:34.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:35:34.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-06-03T06:35:34.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:35:34.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-06-03T06:35:34.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-06-03T06:35:34.612 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509454 2024-06-03T06:35:34.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:35:34.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-06-03T06:35:34.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:35:34.906 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345926 2024-06-03T06:35:34.906 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:35:34.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:35:34.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345926 2024-06-03T06:35:34.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:35:34.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345926 2024-06-03T06:35:34.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2024-06-03T06:35:34.909 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345926 2024-06-03T06:35:34.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:35:35.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345927 -lt 85899345926 2024-06-03T06:35:35.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:35:35.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:35:35.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:35:35.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T06:35:35.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:35:35.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:35:35.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:35:35.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:35:35.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:35:35.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:35:35.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:35:35.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:35:35.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:35:35.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:35:35.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:35:36.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T06:35:36.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:35:36.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:35:36.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-06-03T06:35:36.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-03T06:35:36.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-03T06:35:36.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-06-03T06:35:36.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-03T06:35:36.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-06-03T06:35:36.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-03T06:35:36.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-03T06:35:37.403 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'ecpool' created 2024-06-03T06:35:37.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:35:38.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-06-03T06:35:38.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-06-03T06:35:38.838 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 2 allow_ec_overwrites to true 2024-06-03T06:35:38.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-03T06:35:38.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:35:38.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:35:38.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:35:38.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:35:38.851 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:35:38.851 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:35:38.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:35:38.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:35:38.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:35:38.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:35:38.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:35:38.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:35:38.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:35:38.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:35:38.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:35:39.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:35:39.243 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:35:39.243 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:35:39.243 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-06-03T06:35:39.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:35:39.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:35:39.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:35:39.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836513 2024-06-03T06:35:39.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836513 2024-06-03T06:35:39.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513' 2024-06-03T06:35:39.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:35:39.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:35:39.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-06-03T06:35:39.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-06-03T06:35:39.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672983' 2024-06-03T06:35:39.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:35:39.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:35:39.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-06-03T06:35:39.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-06-03T06:35:39.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672983 2-64424509456' 2024-06-03T06:35:39.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:35:39.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:35:39.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345929 2024-06-03T06:35:39.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345929 2024-06-03T06:35:39.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672983 2-64424509456 3-85899345929' 2024-06-03T06:35:39.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:35:39.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836513 2024-06-03T06:35:39.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:35:39.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:35:39.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836513 2024-06-03T06:35:39.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:35:39.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836513 2024-06-03T06:35:39.676 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836513 2024-06-03T06:35:39.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-06-03T06:35:39.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:35:39.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836513 2024-06-03T06:35:39.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:35:40.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:35:40.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:35:41.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836513 2024-06-03T06:35:41.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:35:41.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-06-03T06:35:41.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:35:41.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:35:41.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-06-03T06:35:41.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:35:41.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-06-03T06:35:41.294 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672983 2024-06-03T06:35:41.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-06-03T06:35:41.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:35:41.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672983 2024-06-03T06:35:41.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:35:41.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-06-03T06:35:41.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:35:41.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:35:41.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-06-03T06:35:41.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:35:41.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-06-03T06:35:41.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-06-03T06:35:41.587 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509456 2024-06-03T06:35:41.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:35:41.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509456 2024-06-03T06:35:41.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:35:41.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345929 2024-06-03T06:35:41.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:35:41.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:35:41.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345929 2024-06-03T06:35:41.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:35:41.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345929 2024-06-03T06:35:41.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2024-06-03T06:35:41.919 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345929 2024-06-03T06:35:41.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:35:42.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345929 -lt 85899345929 2024-06-03T06:35:42.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:35:42.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:35:42.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:35:42.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:35:42.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:35:42.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:35:42.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:35:42.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:35:42.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:35:42.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:35:42.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:35:42.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:35:42.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:35:42.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:35:42.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:35:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:35:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:35:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:35:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-03T06:35:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-06-03T06:35:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-03T06:35:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-06-03T06:35:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-06-03T06:35:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T06:35:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T06:35:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-03T06:35:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T06:35:43.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T06:35:43.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T06:35:43.609 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T06:35:43.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T06:35:44.120 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T06:35:44.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T06:35:44.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T06:35:44.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-03T06:35:44.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-06-03T06:35:44.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-03T06:35:44.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:35:44.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:35:44.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:35:44.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-06-03T06:35:44.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-06-03T06:35:44.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-03T06:35:44.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-06-03T06:35:44.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-06-03T06:35:44.762 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:35:44.765 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T06:35:45.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-06-03T06:35:45.055 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:35:45.055 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:35:45.055 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-06-03T06:35:45.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-06-03T06:35:45.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-06-03T06:35:45.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-06-03T06:35:45.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-06-03T06:35:45.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-06-03T06:35:45.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-06-03T06:35:45.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-03T06:35:45.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-03T06:35:45.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-06-03T06:35:45.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:35:45.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:35:45.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:35:45.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:35:45.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:35:45.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:35:45.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:35:45.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:35:45.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:35:45.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:35:45.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:35:45.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:35:45.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:35:45.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:35:45.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:35:45.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:35:45.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:35:45.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:35:45.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:35:45.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:35:45.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:35:45.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:35:45.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:35:45.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-03T06:35:47.052 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-06-03T06:35:47.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:35:47.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:35:47.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:35:47.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:35:47.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:35:47.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:35:47.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:35:47.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:35:47.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:35:47.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:35:47.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:35:47.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:35:47.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:35:47.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:35:47.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:35:47.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:35:47.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:35:47.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:35:47.575 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:35:47.575 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:35:47.575 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:35:47.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:35:47.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:35:47.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:35:47.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:35:47.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:35:47.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:35:47.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:35:47.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:35:47.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:35:47.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:35:47.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:35:47.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:35:47.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:35:47.578 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.3 2024-06-03T06:35:47.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:35:47.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:35:47.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:35:47.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:35:47.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:35:47.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:35:47.605 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:35:47.604+0000 7fd5637fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:35:47.607 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:35:47.607+0000 7fd5637fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:35:47.608 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:35:47.608+0000 7fd5637fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:35:47.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:35:47.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:35:47.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:35:47.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:35:47.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:35:47.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:47.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:35:47.880 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:35:47.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:47.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:48.126 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:35:48.126+0000 7fd5637fc640 -1 Falling back to public interface 2024-06-03T06:35:48.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:49.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:49.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:49.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:35:49.189 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:35:49.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:49.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:49.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:50.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:50.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:50.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:35:50.490 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:35:50.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:50.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:50.525 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:35:50.524+0000 7fd5637fc640 -1 osd.3 37 log_to_monitors true 2024-06-03T06:35:50.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:35:51.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:35:51.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:35:51.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:35:51.841 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:35:51.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:35:51.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:35:52.155 INFO:tasks.workunit.client.0.smithi039.stderr:osd.3 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [20,37) [v2:127.0.0.1:6826/865342459,v1:127.0.0.1:6827/865342459] [v2:127.0.0.1:6828/865342459,v1:127.0.0.1:6829/865342459] exists,up 6f805989-769d-4078-8009-2c052ac9acc1 2024-06-03T06:35:52.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:35:52.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:35:52.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:35:52.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:35:52.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:35:52.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:35:52.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:35:52.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:35:52.156 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:35:52.156 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:35:52.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:35:52.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:35:52.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:35:52.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:35:52.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:35:52.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:35:52.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:35:52.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:35:52.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:35:52.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:35:52.625 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:35:52.625 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:35:52.626 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-06-03T06:35:52.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:35:52.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:35:52.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:35:52.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836517 2024-06-03T06:35:52.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836517 2024-06-03T06:35:52.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517' 2024-06-03T06:35:52.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:35:52.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:35:52.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672987 2024-06-03T06:35:52.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672987 2024-06-03T06:35:52.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-42949672987' 2024-06-03T06:35:52.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:35:52.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:35:52.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-06-03T06:35:52.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-06-03T06:35:52.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-42949672987 2-64424509460' 2024-06-03T06:35:52.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:35:52.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:35:53.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659138 2024-06-03T06:35:53.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659138 2024-06-03T06:35:53.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-42949672987 2-64424509460 3-176093659138' 2024-06-03T06:35:53.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:35:53.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836517 2024-06-03T06:35:53.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:35:53.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:35:53.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836517 2024-06-03T06:35:53.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:35:53.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836517 2024-06-03T06:35:53.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-06-03T06:35:53.103 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836517 2024-06-03T06:35:53.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:35:53.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836517 -lt 21474836517 2024-06-03T06:35:53.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:35:53.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672987 2024-06-03T06:35:53.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:35:53.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:35:53.391 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672987 2024-06-03T06:35:53.391 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:35:53.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672987 2024-06-03T06:35:53.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-06-03T06:35:53.393 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672987 2024-06-03T06:35:53.393 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:35:53.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672987 2024-06-03T06:35:53.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:35:53.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-06-03T06:35:53.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:35:53.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:35:53.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-06-03T06:35:53.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:35:53.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-06-03T06:35:53.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-06-03T06:35:53.687 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509460 2024-06-03T06:35:53.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:35:53.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509460 2024-06-03T06:35:53.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:35:54.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:35:54.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:35:55.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-06-03T06:35:55.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:35:55.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-176093659138 2024-06-03T06:35:55.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:35:55.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:35:55.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-176093659138 2024-06-03T06:35:55.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:35:55.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659138 2024-06-03T06:35:55.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 176093659138' 2024-06-03T06:35:55.304 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 176093659138 2024-06-03T06:35:55.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:35:55.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659138 -lt 176093659138 2024-06-03T06:35:55.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:35:55.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:35:55.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:35:56.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:35:56.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:35:56.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:35:56.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:35:56.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:35:56.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:35:56.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:35:56.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:35:56.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:35:56.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:35:56.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:35:56.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:35:56.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:35:56.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:35:56.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:35:56.648 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-03T06:35:56.648 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:35:56.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:35:56.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:35:56.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:35:56.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-03T06:35:56.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-03T06:35:56.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:35:56.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:35:56.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:35:56.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:35:56.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:35:56.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:35:57.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:35:37.260414+0000 2024-06-03T06:35:57.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:35:57.499 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:35:57.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:35:37.260414+0000 2024-06-03T06:35:57.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:35:57.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:35:37.260414+0000 2024-06-03T06:35:57.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:35:57.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:35:57.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:35:57.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:35:57.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:35:57.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:35:57.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:35:57.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:35:57.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:35:37.260414+0000 '>' 2024-06-03T06:35:37.260414+0000 2024-06-03T06:35:57.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:35:58.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:35:58.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:35:58.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:35:58.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:35:58.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:35:58.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:35:58.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:35:59.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:35:37.260414+0000 '>' 2024-06-03T06:35:37.260414+0000 2024-06-03T06:35:59.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:36:00.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:36:00.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:36:00.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:36:00.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:36:00.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:36:00.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:36:00.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:36:00.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:35:37.260414+0000 '>' 2024-06-03T06:35:37.260414+0000 2024-06-03T06:36:00.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:36:01.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:36:01.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:36:01.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:36:01.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:36:01.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:36:01.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:36:01.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:36:01.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:35:57.594826+0000 '>' 2024-06-03T06:35:37.260414+0000 2024-06-03T06:36:01.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:36:01.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:36:01.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:36:01.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:36:01.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:36:01.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:36:01.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:36:01.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:36:01.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:36:01.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:36:01.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:36:01.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:36:01.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:36:01.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:36:01.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:36:01.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:36:01.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:36:02.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:36:02.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:36:02.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:36:02.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:36:02.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:36:02.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:36:02.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:36:02.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-03T06:36:03.477 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-03T06:36:03.477 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:36:03.477 INFO:tasks.workunit.client.0.smithi039.stdout:hinfo_key 2024-06-03T06:36:03.477 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:36:03.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:36:03.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:36:03.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:36:03.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:36:03.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:36:03.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:36:03.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:36:03.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:36:03.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:36:03.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:36:03.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:36:03.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:36:03.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:36:03.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:36:03.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:36:03.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:36:03.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:36:03.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:36:03.746 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:36:03.746 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:36:03.746 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:36:03.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:36:03.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:36:03.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:36:03.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:36:03.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:36:03.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:36:03.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:36:03.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:36:03.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:36:03.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:36:03.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:36:03.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:36:03.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:36:03.748 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.3 2024-06-03T06:36:03.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:36:03.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:36:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:36:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:36:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:36:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:36:03.776 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:36:03.776+0000 7f2c0de0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:36:03.778 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:36:03.778+0000 7f2c0de0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:36:03.780 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:36:03.779+0000 7f2c0de0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:36:04.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:36:04.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:36:04.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:36:04.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:36:04.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:36:04.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:04.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:36:04.050 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:36:04.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:04.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:36:04.298 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:36:04.298+0000 7f2c0de0d640 -1 Falling back to public interface 2024-06-03T06:36:04.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:36:05.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:36:05.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:05.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:36:05.341 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:36:05.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:05.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:36:05.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:36:06.584 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:36:06.583+0000 7f2c0de0d640 -1 osd.3 43 log_to_monitors true 2024-06-03T06:36:06.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:36:06.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:06.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:36:06.640 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:36:06.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:06.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:36:07.014 INFO:tasks.workunit.client.0.smithi039.stderr:osd.3 up in weight 1 up_from 47 up_thru 41 down_at 44 last_clean_interval [41,43) [v2:127.0.0.1:6826/3636802376,v1:127.0.0.1:6827/3636802376] [v2:127.0.0.1:6828/3636802376,v1:127.0.0.1:6829/3636802376] exists,up 6f805989-769d-4078-8009-2c052ac9acc1 2024-06-03T06:36:07.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:36:07.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:36:07.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:36:07.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:36:07.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:36:07.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:36:07.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:36:07.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:36:07.015 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:36:07.015 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:36:07.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:36:07.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:36:07.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:36:07.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:36:07.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:36:07.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:36:07.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:36:07.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:36:07.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:36:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:36:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:36:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:36:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-06-03T06:36:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:36:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:07.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:36:07.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836521 2024-06-03T06:36:07.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836521 2024-06-03T06:36:07.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521' 2024-06-03T06:36:07.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:07.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:36:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672991 2024-06-03T06:36:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672991 2024-06-03T06:36:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-42949672991' 2024-06-03T06:36:08.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:08.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:36:08.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-06-03T06:36:08.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-06-03T06:36:08.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-42949672991 2-64424509464' 2024-06-03T06:36:08.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:08.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:36:08.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462914 2024-06-03T06:36:08.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462914 2024-06-03T06:36:08.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-42949672991 2-64424509464 3-201863462914' 2024-06-03T06:36:08.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:08.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836521 2024-06-03T06:36:08.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:08.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:36:08.383 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836521 2024-06-03T06:36:08.383 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:08.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836521 2024-06-03T06:36:08.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-06-03T06:36:08.384 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836521 2024-06-03T06:36:08.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:36:08.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836521 2024-06-03T06:36:08.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:36:09.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:36:09.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:36:10.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836521 -lt 21474836521 2024-06-03T06:36:10.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:10.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672991 2024-06-03T06:36:10.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:10.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:36:10.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672991 2024-06-03T06:36:10.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:10.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672991 2024-06-03T06:36:10.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2024-06-03T06:36:10.028 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672991 2024-06-03T06:36:10.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:36:10.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672991 -lt 42949672991 2024-06-03T06:36:10.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:10.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-06-03T06:36:10.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:10.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:36:10.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-06-03T06:36:10.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:10.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-06-03T06:36:10.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-06-03T06:36:10.344 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509464 2024-06-03T06:36:10.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:36:10.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-06-03T06:36:10.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:10.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-201863462914 2024-06-03T06:36:10.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:10.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:36:10.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-201863462914 2024-06-03T06:36:10.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:10.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462914 2024-06-03T06:36:10.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 201863462914' 2024-06-03T06:36:10.637 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 201863462914 2024-06-03T06:36:10.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:36:10.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462914 -lt 201863462914 2024-06-03T06:36:10.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:36:10.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:10.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:11.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:36:11.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:36:11.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:36:11.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:36:11.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:36:11.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:36:11.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:36:11.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:36:11.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:36:11.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:36:11.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:11.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:11.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-03T06:36:11.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-03T06:36:11.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:36:11.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-03T06:36:11.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:36:11.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:36:12.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:36:12.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:36:12.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:36:12.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:36:12.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:36:12.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:36:12.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:36:12.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:36:12.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:36:12.383 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:36:12.383 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:12.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:12.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-03T06:36:12.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-03T06:36:12.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:36:12.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:36:12.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:36:12.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:36:12.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:36:12.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:36:12.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:36:13.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T06:36:13.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T06:36:13.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T06:36:13.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:36:13.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T06:36:13.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:36:13.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:36:13.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:36:13.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:36:13.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:36:13.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:36:13.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:36:13.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:36:13.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:36:13.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:36:13.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:13.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:13.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:36:13.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:36:13.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:36:13.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:36:14.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:36:14.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-06-03T06:36:14.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-03T06:36:14.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-03T06:36:14.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-06-03T06:36:14.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:36:14.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:36:14.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:36:14.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:36:14.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:36:14.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:36:14.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:36:14.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:36:14.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:36:14.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:36:14.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:36:14.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:36:14.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:36:14.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:36:14.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:36:14.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:36:14.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:36:14.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:36:14.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:36:14.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:36:14.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:36:14.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:36:14.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:36:14.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-03T06:36:16.229 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-03T06:36:16.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:36:16.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:36:16.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:36:16.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:36:16.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:36:16.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:36:16.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:36:16.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:36:16.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:36:16.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:36:16.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:36:16.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:36:16.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:36:16.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:36:16.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:36:16.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:36:16.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:36:16.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:36:16.748 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:36:16.748 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:36:16.748 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:36:16.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:36:16.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:36:16.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:36:16.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:36:16.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:36:16.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:36:16.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:36:16.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:36:16.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:36:16.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:36:16.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:36:16.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:36:16.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:36:16.749 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:36:16.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:36:16.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:36:16.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:36:16.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:36:16.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:36:16.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:36:16.778 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:36:16.777+0000 7f64cd413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:36:16.781 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:36:16.779+0000 7f64cd413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:36:16.782 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:36:16.781+0000 7f64cd413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:36:17.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:36:17.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:36:17.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:36:17.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:36:17.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:36:17.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:17.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:36:17.056 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:36:17.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:17.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:36:17.300 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:36:17.299+0000 7f64cd413640 -1 Falling back to public interface 2024-06-03T06:36:17.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:36:18.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:36:18.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:18.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:36:18.374 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:36:18.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:18.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:36:18.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:36:19.640 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:36:19.638+0000 7f64cd413640 -1 osd.1 49 log_to_monitors true 2024-06-03T06:36:19.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:36:19.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:19.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:36:19.673 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:36:19.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:19.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:36:19.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:36:20.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:36:20.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:20.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:36:20.980 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:36:20.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:20.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:36:21.479 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 54 up_thru 54 down_at 50 last_clean_interval [10,49) [v2:127.0.0.1:6810/3281899408,v1:127.0.0.1:6811/3281899408] [v2:127.0.0.1:6812/3281899408,v1:127.0.0.1:6813/3281899408] exists,up c1bf0b94-99bf-4fb0-823c-5047aa4d2dd5 2024-06-03T06:36:21.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:36:21.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:36:21.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:36:21.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:36:21.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:36:21.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:36:21.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:36:21.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:36:21.481 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:36:21.481 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:36:21.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:36:21.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:36:21.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:36:21.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:36:21.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:36:21.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:36:21.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:36:21.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:36:21.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:36:21.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:36:21.881 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:36:21.881 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:36:21.881 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-06-03T06:36:21.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:36:21.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:21.882 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:36:22.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836525 2024-06-03T06:36:22.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836525 2024-06-03T06:36:22.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525' 2024-06-03T06:36:22.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:22.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:36:22.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233986 2024-06-03T06:36:22.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233986 2024-06-03T06:36:22.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-231928233986' 2024-06-03T06:36:22.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:22.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:36:22.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509468 2024-06-03T06:36:22.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509468 2024-06-03T06:36:22.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-231928233986 2-64424509468' 2024-06-03T06:36:22.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:22.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:36:22.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462918 2024-06-03T06:36:22.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462918 2024-06-03T06:36:22.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-231928233986 2-64424509468 3-201863462918' 2024-06-03T06:36:22.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:22.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836525 2024-06-03T06:36:22.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:22.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:36:22.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836525 2024-06-03T06:36:22.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:22.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836525 2024-06-03T06:36:22.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-06-03T06:36:22.360 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836525 2024-06-03T06:36:22.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:36:22.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836525 2024-06-03T06:36:22.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:36:23.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:36:23.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:36:23.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836525 2024-06-03T06:36:23.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:23.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-231928233986 2024-06-03T06:36:23.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:23.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:36:23.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-231928233986 2024-06-03T06:36:23.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:23.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233986 2024-06-03T06:36:23.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 231928233986' 2024-06-03T06:36:23.998 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 231928233986 2024-06-03T06:36:23.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:36:24.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233986 -lt 231928233986 2024-06-03T06:36:24.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:24.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509468 2024-06-03T06:36:24.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:24.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:36:24.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509468 2024-06-03T06:36:24.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:24.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509468 2024-06-03T06:36:24.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-06-03T06:36:24.295 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509468 2024-06-03T06:36:24.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:36:24.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509468 -lt 64424509468 2024-06-03T06:36:24.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:24.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-201863462918 2024-06-03T06:36:24.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:24.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:36:24.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-201863462918 2024-06-03T06:36:24.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:24.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462918 2024-06-03T06:36:24.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 201863462918' 2024-06-03T06:36:24.628 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 201863462918 2024-06-03T06:36:24.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:36:24.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462918 -lt 201863462918 2024-06-03T06:36:24.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:36:24.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:24.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:25.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:36:25.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:36:25.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:36:25.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:36:25.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:36:25.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:36:25.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:36:25.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:36:25.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:36:25.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:36:25.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:25.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:25.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:36:25.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:36:25.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:36:25.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-03T06:36:25.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:36:25.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:36:25.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:36:25.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:36:26.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-03T06:36:26.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-03T06:36:26.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:36:26.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:36:26.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:36:26.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:36:26.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:36:26.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:36:26.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:35:57.594826+0000 2024-06-03T06:36:26.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:36:26.784 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:36:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:35:57.594826+0000 2024-06-03T06:36:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:36:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:35:57.594826+0000 2024-06-03T06:36:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:36:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:36:26.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:36:26.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:36:26.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:36:26.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:36:26.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:36:26.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:36:27.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:35:57.594826+0000 '>' 2024-06-03T06:35:57.594826+0000 2024-06-03T06:36:27.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:36:28.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:36:28.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:36:28.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:36:28.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:36:28.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:36:28.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:36:28.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:36:28.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:35:57.594826+0000 '>' 2024-06-03T06:35:57.594826+0000 2024-06-03T06:36:28.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:36:29.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:36:29.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:36:29.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:36:29.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:36:29.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:36:29.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:36:29.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:36:29.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:35:57.594826+0000 '>' 2024-06-03T06:35:57.594826+0000 2024-06-03T06:36:29.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:36:30.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:36:30.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:36:30.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:36:30.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:36:30.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:36:30.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:36:30.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:36:30.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:35:57.594826+0000 '>' 2024-06-03T06:35:57.594826+0000 2024-06-03T06:36:30.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:36:31.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:36:31.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:36:31.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:36:31.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:36:31.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:36:31.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:36:31.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:36:32.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:35:57.594826+0000 '>' 2024-06-03T06:35:57.594826+0000 2024-06-03T06:36:32.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:36:33.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:36:33.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:36:33.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:36:33.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:36:33.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:36:33.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:36:33.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:36:33.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:36:27.464967+0000 '>' 2024-06-03T06:35:57.594826+0000 2024-06-03T06:36:33.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:36:33.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:36:33.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:36:33.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:36:33.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:36:33.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:36:33.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:36:33.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:36:33.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:36:33.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:36:33.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:36:33.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:36:33.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:36:33.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:36:33.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:36:33.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:36:33.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:36:33.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:36:33.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:36:33.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:36:33.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:36:33.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:36:33.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:36:33.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:36:33.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-03T06:36:35.413 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-03T06:36:35.414 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:36:35.414 INFO:tasks.workunit.client.0.smithi039.stdout:hinfo_key 2024-06-03T06:36:35.414 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:36:35.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:36:35.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:36:35.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:36:35.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:36:35.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:36:35.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:36:35.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:36:35.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:36:35.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:36:35.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:36:35.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:36:35.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:36:35.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:36:35.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:36:35.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:36:35.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:36:35.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:36:35.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:36:35.682 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:36:35.682 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:36:35.682 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:36:35.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:36:35.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:36:35.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:36:35.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:36:35.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:36:35.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:36:35.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:36:35.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:36:35.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:36:35.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:36:35.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:36:35.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:36:35.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:36:35.684 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.3 2024-06-03T06:36:35.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:36:35.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:36:35.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:36:35.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:36:35.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:36:35.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:36:35.713 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:36:35.713+0000 7fa7e0531640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:36:35.716 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:36:35.715+0000 7fa7e0531640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:36:35.717 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:36:35.716+0000 7fa7e0531640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:36:36.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:36:36.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:36:36.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:36:36.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:36:36.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:36:36.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:36.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:36:36.010 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:36:36.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:36.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:36:36.234 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:36:36.234+0000 7fa7e0531640 -1 Falling back to public interface 2024-06-03T06:36:36.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:36:37.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:36:37.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:37.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:36:37.304 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:36:37.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:37.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:36:37.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:36:38.438 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:36:38.438+0000 7fa7e0531640 -1 osd.3 56 log_to_monitors true 2024-06-03T06:36:38.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:36:38.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:38.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:36:38.607 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:36:38.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:38.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:36:38.943 INFO:tasks.workunit.client.0.smithi039.stderr:osd.3 up in weight 1 up_from 60 up_thru 54 down_at 57 last_clean_interval [47,56) [v2:127.0.0.1:6826/3055334854,v1:127.0.0.1:6827/3055334854] [v2:127.0.0.1:6828/3055334854,v1:127.0.0.1:6829/3055334854] exists,up 6f805989-769d-4078-8009-2c052ac9acc1 2024-06-03T06:36:38.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:36:38.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:36:38.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:36:38.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:36:38.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:36:38.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:36:38.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:36:38.944 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:36:38.944 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:36:38.945 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:36:38.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:36:38.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:36:38.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:36:39.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:36:39.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:36:39.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:36:39.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:36:39.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:36:39.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:36:39.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:36:39.384 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:36:39.384 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:36:39.385 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-06-03T06:36:39.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:36:39.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:39.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:36:39.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836529 2024-06-03T06:36:39.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836529 2024-06-03T06:36:39.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529' 2024-06-03T06:36:39.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:39.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:36:39.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233990 2024-06-03T06:36:39.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233990 2024-06-03T06:36:39.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-231928233990' 2024-06-03T06:36:39.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:39.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:36:40.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509472 2024-06-03T06:36:40.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509472 2024-06-03T06:36:40.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-231928233990 2-64424509472' 2024-06-03T06:36:40.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:40.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:36:40.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037762 2024-06-03T06:36:40.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037762 2024-06-03T06:36:40.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-231928233990 2-64424509472 3-257698037762' 2024-06-03T06:36:40.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:40.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836529 2024-06-03T06:36:40.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:40.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:36:40.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836529 2024-06-03T06:36:40.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:40.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836529 2024-06-03T06:36:40.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-06-03T06:36:40.159 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836529 2024-06-03T06:36:40.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:36:40.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836529 2024-06-03T06:36:40.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:36:41.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:36:41.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:36:41.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836529 2024-06-03T06:36:41.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:41.798 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-231928233990 2024-06-03T06:36:41.798 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:41.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:36:41.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-231928233990 2024-06-03T06:36:41.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:41.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233990 2024-06-03T06:36:41.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 231928233990' 2024-06-03T06:36:41.801 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 231928233990 2024-06-03T06:36:41.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:36:42.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233991 -lt 231928233990 2024-06-03T06:36:42.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:42.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509472 2024-06-03T06:36:42.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:42.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:36:42.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509472 2024-06-03T06:36:42.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:42.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509472 2024-06-03T06:36:42.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-06-03T06:36:42.164 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509472 2024-06-03T06:36:42.164 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:36:42.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509473 -lt 64424509472 2024-06-03T06:36:42.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:42.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037762 2024-06-03T06:36:42.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:42.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:36:42.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037762 2024-06-03T06:36:42.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:42.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037762 2024-06-03T06:36:42.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037762' 2024-06-03T06:36:42.457 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 257698037762 2024-06-03T06:36:42.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:36:42.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037762 -lt 257698037762 2024-06-03T06:36:42.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:36:42.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:42.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:43.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:36:43.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:36:43.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:36:43.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:36:43.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:36:43.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:36:43.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:36:43.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:36:43.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:36:43.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:36:43.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:43.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:43.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:36:43.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:36:43.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:36:43.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:36:43.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:36:43.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-06-03T06:36:43.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-03T06:36:43.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-03T06:36:43.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-06-03T06:36:43.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-06-03T06:36:43.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-03T06:36:43.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:36:43.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:36:43.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:36:43.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 191555"' 2024-06-03T06:36:43.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 191555' 2024-06-03T06:36:43.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-06-03T06:36:43.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:36:43.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:36:43.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 191557"' 2024-06-03T06:36:43.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:36:43.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 191557' 2024-06-03T06:36:43.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-03T06:36:43.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 191555 191557' 2024-06-03T06:36:43.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:36:43.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:36:43.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 191555 2024-06-03T06:36:43.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/191558: /' 2024-06-03T06:36:43.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-06-03T06:36:43.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/191560: /' 2024-06-03T06:36:46.765 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:36:46.765 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:36:46.765 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:36:46.765 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:36:46.765 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:36:46.766 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:36:46.766 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:36:46.766 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:36:46.766 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:36:46.766 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:36:46.766 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:36:46.766 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:36:46.766 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:36:46.766 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:36:46.766 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:36:46.766 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:36:46.766 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:36:46.767 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:36:46.767 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:36:46.767 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:36:46.767 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:36:46.767 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:36:46.767 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-03T06:36:46.767 INFO:tasks.workunit.client.0.smithi039.stderr:191558: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-03T06:36:46.767 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:36:46.767 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:36:46.767 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:36:46.767 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:36:46.767 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:36:46.767 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:36:46.768 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:36:46.768 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:36:46.769 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:36:46.769 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:36:46.769 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:36:46.769 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:36:46.769 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:36:46.769 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:36:46.769 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:36:46.769 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:36:46.769 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:36:46.769 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:36:46.769 INFO:tasks.workunit.client.0.smithi039.stderr:191558: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:36:46.770 INFO:tasks.workunit.client.0.smithi039.stderr:191558: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:36:46.770 INFO:tasks.workunit.client.0.smithi039.stderr:191558: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:36:46.770 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:36:46.770 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:36:46.770 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:36:46.770 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:36:46.770 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:36:46.770 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:36:46.770 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:36:46.770 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:36:46.770 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:36:46.770 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:36:46.770 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:36:46.771 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:36:46.771 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:36:46.771 INFO:tasks.workunit.client.0.smithi039.stderr:191558: start osd.1 2024-06-03T06:36:46.771 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:36:46.870 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:36:46.870 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:36:46.870 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-03T06:36:46.870 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:36:46.870 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-06-03T06:36:46.870 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:36:46.870 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:36:46.870 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-03T06:36:46.870 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:36:46.871 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-03T06:36:46.871 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:36:46.871 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:36:46.871 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:36:46.871 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:36:46.871 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:36:46.871 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:36:46.871 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-06-03T06:36:46.871 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:36:46.871 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:36:46.871 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-03T06:36:46.871 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:36:46.871 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:36:46.871 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-06-03T06:36:46.872 INFO:tasks.workunit.client.0.smithi039.stderr:191560: remove 2#2:eb822e21:::SOMETHING:head# 2024-06-03T06:36:46.872 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-03T06:36:46.872 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:36:46.872 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:36:46.872 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-03T06:36:46.872 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:36:46.872 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:36:46.872 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:36:46.872 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:36:46.873 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:36:46.873 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:36:46.873 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:36:46.873 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:36:46.873 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:36:46.873 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:36:46.873 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:36:46.873 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:36:46.873 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:36:46.874 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:36:46.874 INFO:tasks.workunit.client.0.smithi039.stderr:191560: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:36:46.874 INFO:tasks.workunit.client.0.smithi039.stderr:191560: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:36:46.874 INFO:tasks.workunit.client.0.smithi039.stderr:191560: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:36:46.874 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:36:46.874 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:36:46.874 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:36:46.874 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:36:46.874 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:36:46.874 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:36:46.874 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:36:46.874 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:36:46.874 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:36:46.875 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:36:46.875 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:36:46.875 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:36:46.875 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-03T06:36:46.875 INFO:tasks.workunit.client.0.smithi039.stderr:191560: start osd.2 2024-06-03T06:36:46.875 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:36:51.324 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:36:51.324 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:36:51.324 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:36:51.324 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:36:51.324 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:36:51.324 INFO:tasks.workunit.client.0.smithi039.stderr:191558: 2024-06-03T06:36:46.799+0000 7f0b51a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:36:51.324 INFO:tasks.workunit.client.0.smithi039.stderr:191558: 2024-06-03T06:36:46.801+0000 7f0b51a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:36:51.325 INFO:tasks.workunit.client.0.smithi039.stderr:191558: 2024-06-03T06:36:46.803+0000 7f0b51a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:36:51.325 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:36:51.325 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:36:51.325 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:36:51.325 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:36:51.325 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:36:51.325 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:51.325 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:36:51.325 INFO:tasks.workunit.client.0.smithi039.stderr:191558: 0 2024-06-03T06:36:51.325 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:51.326 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:36:51.326 INFO:tasks.workunit.client.0.smithi039.stderr:191558: 2024-06-03T06:36:47.319+0000 7f0b51a0f640 -1 Falling back to public interface 2024-06-03T06:36:51.326 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:36:51.326 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:36:51.326 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:51.326 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:36:51.326 INFO:tasks.workunit.client.0.smithi039.stderr:191558: 1 2024-06-03T06:36:51.326 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:51.326 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:36:51.326 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:36:51.326 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:36:51.327 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:51.327 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:36:51.327 INFO:tasks.workunit.client.0.smithi039.stderr:191558: 2 2024-06-03T06:36:51.327 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:51.327 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:36:51.327 INFO:tasks.workunit.client.0.smithi039.stderr:191558: 2024-06-03T06:36:49.974+0000 7f0b51a0f640 -1 osd.1 62 log_to_monitors true 2024-06-03T06:36:51.327 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:36:51.327 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:36:51.327 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:51.327 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:36:51.327 INFO:tasks.workunit.client.0.smithi039.stderr:191558: 3 2024-06-03T06:36:51.327 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:51.328 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:36:51.570 INFO:tasks.workunit.client.0.smithi039.stderr:191558: osd.1 up in weight 1 up_from 67 up_thru 67 down_at 63 last_191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-03T06:36:51.570 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-03T06:36:51.570 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:36:51.570 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:36:51.570 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:36:51.570 INFO:tasks.workunit.client.0.smithi039.stderr:191560: 2024-06-03T06:36:46.900+0000 7f366bc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:36:51.571 INFO:tasks.workunit.client.0.smithi039.stderr:191560: 2024-06-03T06:36:46.903+0000 7f366bc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:36:51.571 INFO:tasks.workunit.client.0.smithi039.stderr:191560: 2024-06-03T06:36:46.904+0000 7f366bc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:36:51.571 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-03T06:36:51.571 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:36:51.571 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:36:51.571 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:36:51.571 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:36:51.571 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:51.571 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:36:51.571 INFO:tasks.workunit.client.0.smithi039.stderr:191560: 0 2024-06-03T06:36:51.571 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:51.571 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:36:51.571 INFO:tasks.workunit.client.0.smithi039.stderr:191560: 2024-06-03T06:36:47.423+0000 7f366bc0d640 -1 Falling back to public interface 2024-06-03T06:36:51.572 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:36:51.572 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:36:51.572 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:51.572 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:36:51.572 INFO:tasks.workunit.client.0.smithi039.stderr:191560: 1 2024-06-03T06:36:51.572 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:51.572 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:36:51.572 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:36:51.572 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:36:51.572 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:51.572 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:36:51.572 INFO:tasks.workunit.client.0.smithi039.stderr:191560: 2 2024-06-03T06:36:51.572 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:51.573 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:36:51.573 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:36:51.573 INFO:tasks.workunit.client.0.smithi039.stderr:191560: 2024-06-03T06:36:50.364+0000 7f366bc0d640 -1 osd.2 62 log_to_monitors true 2024-06-03T06:36:51.573 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:36:51.573 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:36:51.573 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:36:51.573 INFO:tasks.workunit.client.0.smithi039.stderr:191560: 3 2024-06-03T06:36:51.573 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:36:51.573 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:36:52.149 INFO:tasks.workunit.client.0.smithi039.stderr:191560: osd.2 up in weight 1 up_from 69 up_thru 50 down_at 63 last_clean_interval [54,62) [v2:127.0.0.1:6810/3614149264,v1:127.0.0.1:6811/3614149264] [v2:127.0.0.1:6812/3614149264,v1:127.0.0.1:6813/3614149264] exists,up c1bf0b94-99bf-4fb0-823c-5047aa4d2dd5 2024-06-03T06:36:52.150 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:36:52.150 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:36:52.150 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:36:52.150 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:36:52.150 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:36:52.150 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:36:52.150 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:36:52.150 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:36:52.150 INFO:tasks.workunit.client.0.smithi039.stderr:191558: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:36:52.150 INFO:tasks.workunit.client.0.smithi039.stderr:191558: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:36:52.150 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:36:52.150 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:36:52.151 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:36:52.151 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:36:52.151 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:36:52.151 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:36:52.151 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:36:52.151 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:36:52.151 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:36:52.151 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:36:52.151 INFO:tasks.workunit.client.0.smithi039.stderr:191558: 1 2024-06-03T06:36:52.151 INFO:tasks.workunit.client.0.smithi039.stderr:191558: 2 2024-06-03T06:36:52.151 INFO:tasks.workunit.client.0.smithi039.stderr:191558: 3' 2024-06-03T06:36:52.151 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:36:52.151 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:52.152 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:36:52.152 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836533 2024-06-03T06:36:52.152 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836533 2024-06-03T06:36:52.152 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533' 2024-06-03T06:36:52.152 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:52.152 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:36:52.152 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-06-03T06:36:52.152 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-06-03T06:36:52.152 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-287762808834' 2024-06-03T06:36:52.152 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:52.503 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [15,62) [v2:127.0.0.1:6818/3838829828,v1:127.0.0.1:6819/3838829828] [v2:127.0.0.1:6820/3838829828,v1:127.0.0.1:6821/3838829828] exists,up 5cd6cf24-6678-4c6e-9721-cd271fc1f0af 2024-06-03T06:36:52.503 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:36:52.503 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:36:52.504 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:36:52.504 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:36:52.504 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:36:52.504 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:36:52.504 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:36:52.504 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:36:52.504 INFO:tasks.workunit.client.0.smithi039.stderr:191560: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:36:52.504 INFO:tasks.workunit.client.0.smithi039.stderr:191560: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:36:52.504 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:36:52.504 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:36:52.504 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:36:52.504 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:36:52.504 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:36:52.504 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:36:52.504 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:36:52.505 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:36:52.505 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:36:52.505 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:36:52.505 INFO:tasks.workunit.client.0.smithi039.stderr:191560: 1 2024-06-03T06:36:52.505 INFO:tasks.workunit.client.0.smithi039.stderr:191560: 2 2024-06-03T06:36:52.505 INFO:tasks.workunit.client.0.smithi039.stderr:191560: 3' 2024-06-03T06:36:52.505 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:36:52.505 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:52.505 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:36:52.505 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836534 2024-06-03T06:36:52.505 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836534 2024-06-03T06:36:52.505 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534' 2024-06-03T06:36:52.505 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:52.506 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:36:52.506 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808835 2024-06-03T06:36:52.506 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808835 2024-06-03T06:36:52.506 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-287762808835' 2024-06-03T06:36:52.506 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:53.457 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:36:53.457 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743427 2024-06-03T06:36:53.457 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743427 2024-06-03T06:36:53.457 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-287762808835 2-296352743427' 2024-06-03T06:36:53.457 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:53.457 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:36:53.457 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037766 2024-06-03T06:36:53.457 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037766 2024-06-03T06:36:53.457 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-287762808835 2-296352743427 3-257698037766' 2024-06-03T06:36:53.457 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:53.457 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836534 2024-06-03T06:36:53.458 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:53.458 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:36:53.458 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836534 2024-06-03T06:36:53.458 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:53.458 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836534 2024-06-03T06:36:53.458 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-06-03T06:36:53.458 INFO:tasks.workunit.client.0.smithi039.stderr:191560: waiting osd.0 seq 21474836534 2024-06-03T06:36:53.458 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:36:53.458 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836534 2024-06-03T06:36:53.458 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:53.458 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808835 2024-06-03T06:36:53.458 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:53.459 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:36:53.459 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808835 2024-06-03T06:36:53.459 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:53.459 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808835 2024-06-03T06:36:53.459 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808835' 2024-06-03T06:36:53.459 INFO:tasks.workunit.client.0.smithi039.stderr:191560: waiting osd.1 seq 287762808835 2024-06-03T06:36:53.459 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:36:53.459 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808835 2024-06-03T06:36:53.459 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:53.459 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-296352743427 2024-06-03T06:36:53.459 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:53.459 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:36:53.459 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-296352743427 2024-06-03T06:36:54.447 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtell osd.2 flush_pg_stats 2024-06-03T06:36:54.447 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743426 2024-06-03T06:36:54.447 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743426 2024-06-03T06:36:54.448 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-287762808834 2-296352743426' 2024-06-03T06:36:54.448 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:36:54.448 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:36:54.448 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037765 2024-06-03T06:36:54.448 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037765 2024-06-03T06:36:54.448 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-287762808834 2-296352743426 3-257698037765' 2024-06-03T06:36:54.448 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:54.448 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836533 2024-06-03T06:36:54.448 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:54.448 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:36:54.449 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836533 2024-06-03T06:36:54.449 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:54.449 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836533 2024-06-03T06:36:54.449 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-06-03T06:36:54.449 INFO:tasks.workunit.client.0.smithi039.stderr:191558: waiting osd.0 seq 21474836533 2024-06-03T06:36:54.449 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:36:54.449 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836533 2024-06-03T06:36:54.449 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:36:54.449 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:36:54.449 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:36:54.450 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836533 2024-06-03T06:36:54.450 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:54.450 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808834 2024-06-03T06:36:54.450 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:54.450 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:36:54.450 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808834 2024-06-03T06:36:54.450 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:54.450 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-06-03T06:36:54.450 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2024-06-03T06:36:54.450 INFO:tasks.workunit.client.0.smithi039.stderr:191558: waiting osd.1 seq 287762808834 2024-06-03T06:36:54.450 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:36:54.450 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808834 2024-06-03T06:36:54.451 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:1915test/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743427 2024-06-03T06:36:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 296352743427' 2024-06-03T06:36:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:191560: waiting osd.2 seq 296352743427 2024-06-03T06:36:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:36:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743427 -lt 296352743427 2024-06-03T06:36:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037766 2024-06-03T06:36:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:36:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037766 2024-06-03T06:36:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037766 2024-06-03T06:36:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037766' 2024-06-03T06:36:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:191560: waiting osd.3 seq 257698037766 2024-06-03T06:36:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:36:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037766 -lt 257698037766 2024-06-03T06:36:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:36:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:36:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:36:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:36:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:36:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:36:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:36:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:36:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:36:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T06:36:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:36:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-03T06:36:55.098 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-03T06:36:55.791 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalon58: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-296352743426 2024-06-03T06:36:55.791 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:55.791 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:36:55.791 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-296352743426 2024-06-03T06:36:55.791 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:55.792 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743426 2024-06-03T06:36:55.792 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 296352743426' 2024-06-03T06:36:55.792 INFO:tasks.workunit.client.0.smithi039.stderr:191558: waiting osd.2 seq 296352743426 2024-06-03T06:36:55.792 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:36:55.792 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743427 -lt 296352743426 2024-06-03T06:36:55.792 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:36:55.792 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037765 2024-06-03T06:36:55.792 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:36:55.792 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:36:55.792 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037765 2024-06-03T06:36:55.793 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:36:55.793 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037765 2024-06-03T06:36:55.793 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037765' 2024-06-03T06:36:55.793 INFO:tasks.workunit.client.0.smithi039.stderr:191558: waiting osd.3 seq 257698037765 2024-06-03T06:36:55.793 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:36:55.793 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037767 -lt 257698037765 2024-06-03T06:36:55.793 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:36:55.793 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:55.793 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:55.793 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:36:55.793 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:36:55.794 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:36:55.794 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:36:55.794 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:36:55.794 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:36:55.794 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:36:55.794 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:36:55.794 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T06:36:55.794 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:36:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.ce/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:36:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-03T06:36:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:36:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:36:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:36:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:36:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:36:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:36:56.501 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:36:56.501 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:36:56.501 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:36:56.501 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:36:56.501 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T06:36:56.501 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:36:56.501 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:56.501 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:56.501 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-03T06:36:56.501 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T06:36:56.501 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:36:56.501 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:36:56.502 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:36:56.502 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:36:56.502 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:36:56.502 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:36:56.502 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:36:56.502 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T06:36:56.502 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T06:36:56.502 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T06:36:56.502 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:36:56.502 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T06:36:56.502 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:36:57.310 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1lient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:57.310 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:57.310 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-03T06:36:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-03T06:36:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:36:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-03T06:36:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:36:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:36:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:36:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:36:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:36:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:36:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:36:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:36:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:36:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:36:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T06:36:57.312 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:36:57.312 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:57.312 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:57.312 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-03T06:36:57.312 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T06:36:57.312 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:36:57.312 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:36:57.312 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:36:57.312 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:36:57.312 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:36:57.312 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:36:57.312 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:36:57.312 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T06:36:57.312 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T06:36:57.936 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/653: wait_for_clean: true 2024-06-03T06:36:57.936 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:36:57.936 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:36:57.936 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:36:57.936 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:36:57.936 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:36:57.936 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:36:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T06:36:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:36:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-03T06:36:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T06:36:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:36:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:36:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:36:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:36:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:36:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:36:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:36:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T06:36:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T06:36:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-03T06:36:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:36:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-03T06:36:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:36:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:36:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:36:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:36:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:36:58.210 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T06:36:58.210 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:36:58.210 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T06:36:58.210 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:36:58.210 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:36:58.210 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:36:58.210 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:36:58.210 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:36:58.210 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:36:58.211 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:36:58.211 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:36:58.211 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:36:58.211 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:36:58.211 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:58.211 INFO:tasks.workunit.client.0.smithi039.stderr:191558: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:58.211 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:36:58.211 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:36:58.211 INFO:tasks.workunit.client.0.smithi039.stderr:191558: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:36:58.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:36:58.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:36:58.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 191557 2024-06-03T06:36:58.632 INFO:tasks.workunit.client.0.smithi039.stderr:select(contains("stale") | not)' 2024-06-03T06:36:58.632 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:36:58.632 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:36:58.633 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:36:58.633 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:36:58.633 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:36:58.633 INFO:tasks.workunit.client.0.smithi039.stderr:191560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:36:58.633 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:36:58.633 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:36:58.633 INFO:tasks.workunit.client.0.smithi039.stderr:191560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:36:58.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:36:58.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:36:58.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:36:58.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:36:58.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-03T06:36:58.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:36:58.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-03T06:36:58.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:36:58.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:36:58.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:36:58.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:36:58.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-03T06:36:58.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-03T06:36:58.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:36:58.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:36:58.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:36:58.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:36:58.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:36:58.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:36:59.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:36:27.464967+0000 2024-06-03T06:36:59.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:36:59.503 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:36:59.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:36:27.464967+0000 2024-06-03T06:36:59.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:36:59.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:36:27.464967+0000 2024-06-03T06:36:59.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:36:59.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:36:59.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:36:59.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:36:59.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:36:59.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:36:59.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:36:59.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:36:59.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:36:27.464967+0000 '>' 2024-06-03T06:36:27.464967+0000 2024-06-03T06:36:59.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:37:00.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:37:00.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:37:00.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:37:00.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:37:00.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:37:00.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:37:00.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:37:01.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:36:27.464967+0000 '>' 2024-06-03T06:36:27.464967+0000 2024-06-03T06:37:01.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:37:02.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:37:02.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:37:02.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:37:02.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:37:02.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:37:02.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:37:02.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:37:02.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:36:27.464967+0000 '>' 2024-06-03T06:36:27.464967+0000 2024-06-03T06:37:02.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:37:03.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:37:03.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:37:03.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:37:03.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:37:03.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:37:03.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:37:03.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:37:03.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:36:27.464967+0000 '>' 2024-06-03T06:36:27.464967+0000 2024-06-03T06:37:03.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:37:04.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:37:04.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:37:04.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:37:04.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:37:04.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:37:04.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:37:04.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:37:05.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:37:00.485422+0000 '>' 2024-06-03T06:36:27.464967+0000 2024-06-03T06:37:05.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:37:05.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-03T06:37:05.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:37:05.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:37:05.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:37:05.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 194703"' 2024-06-03T06:37:05.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 194703' 2024-06-03T06:37:05.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-03T06:37:05.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:37:05.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:37:05.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:37:05.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 194705"' 2024-06-03T06:37:05.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 194705' 2024-06-03T06:37:05.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/194706: /' 2024-06-03T06:37:05.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-03T06:37:05.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 194703 194705' 2024-06-03T06:37:05.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:37:05.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:37:05.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 194703 2024-06-03T06:37:05.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-03T06:37:05.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/194708: /' 2024-06-03T06:37:07.632 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:37:07.633 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:37:07.633 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:37:07.633 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:37:07.633 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:37:07.633 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:37:07.633 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:37:07.633 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:37:07.633 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:37:07.633 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:37:07.633 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:37:07.633 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:37:07.633 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:37:07.634 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:37:07.634 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:37:07.634 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:37:07.634 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:37:07.634 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:37:07.634 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:37:07.634 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:37:07.634 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:37:07.634 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:37:07.634 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-03T06:37:07.634 INFO:tasks.workunit.client.0.smithi039.stderr:194706: _ 2024-06-03T06:37:07.634 INFO:tasks.workunit.client.0.smithi039.stderr:194706: hinfo_key 2024-06-03T06:37:07.634 INFO:tasks.workunit.client.0.smithi039.stderr:194706: snapset 2024-06-03T06:37:07.635 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:37:07.635 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:37:07.635 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:37:07.635 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:37:07.635 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:37:07.635 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:37:07.635 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:37:07.635 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:37:07.636 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:37:07.636 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:37:07.636 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:37:07.636 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:37:07.636 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:37:07.636 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:37:07.636 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:37:07.637 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:37:07.637 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:37:07.637 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:37:07.637 INFO:tasks.workunit.client.0.smithi039.stderr:194706: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:37:07.637 INFO:tasks.workunit.client.0.smithi039.stderr:194706: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:37:07.637 INFO:tasks.workunit.client.0.smithi039.stderr:194706: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:37:07.637 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:07.637 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:07.637 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:37:07.637 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:37:07.637 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:37:07.637 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:37:07.637 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:37:07.637 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:37:07.638 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:37:07.638 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:37:07.638 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:37:07.638 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:37:07.638 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:37:07.638 INFO:tasks.workunit.client.0.smithi039.stderr:194706: start osd.1 2024-06-03T06:37:07.665 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:37:07.665 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:37:07.665 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-03T06:37:07.665 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:37:07.665 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-03T06:37:07.665 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:37:07.665 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:37:07.665 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-03T06:37:07.665 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:37:07.665 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-03T06:37:07.665 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:37:07.666 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:37:07.666 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:37:07.666 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:37:07.666 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:37:07.666 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:37:07.666 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-03T06:37:07.666 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:37:07.666 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:37:07.666 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-03T06:37:07.666 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:37:07.666 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:37:07.666 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-06-03T06:37:07.666 INFO:tasks.workunit.client.0.smithi039.stderr:194708: _ 2024-06-03T06:37:07.667 INFO:tasks.workunit.client.0.smithi039.stderr:194708: hinfo_key 2024-06-03T06:37:07.667 INFO:tasks.workunit.client.0.smithi039.stderr:194708: snapset 2024-06-03T06:37:07.667 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-03T06:37:07.667 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:37:07.667 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:37:07.667 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-03T06:37:07.667 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:37:07.667 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:37:07.667 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:37:07.667 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:37:07.668 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:37:07.668 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:37:07.668 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:37:07.668 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:37:07.668 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:37:07.668 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:37:07.668 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:37:07.668 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:37:07.668 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:37:07.668 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:37:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:194708: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:37:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:194708: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:37:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:194708: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:37:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:37:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:37:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:37:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:37:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:37:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:37:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:37:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:37:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:37:07.670 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:37:07.670 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-03T06:37:07.670 INFO:tasks.workunit.client.0.smithi039.stderr:194708: start osd.2 2024-06-03T06:37:12.232 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:37:12.232 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:37:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:37:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:37:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:37:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:37:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:194706: 2024-06-03T06:37:07.668+0000 7f2680419640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:194706: 2024-06-03T06:37:07.672+0000 7f2680419640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:194706: 2024-06-03T06:37:07.673+0000 7f2680419640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:37:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:37:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:37:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:37:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:37:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:37:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:194706: 0 2024-06-03T06:37:12.234 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:12.234 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:37:12.234 INFO:tasks.workunit.client.0.smithi039.stderr:194706: 2024-06-03T06:37:08.187+0000 7f2680419640 -1 Falling back to public interface 2024-06-03T06:37:12.234 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:12.234 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:12.234 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:12.234 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:37:12.234 INFO:tasks.workunit.client.0.smithi039.stderr:194706: 1 2024-06-03T06:37:12.234 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:12.234 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:37:12.234 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:12.234 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:12.234 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:12.235 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:37:12.235 INFO:tasks.workunit.client.0.smithi039.stderr:194706: 2 2024-06-03T06:37:12.235 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:12.235 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:37:12.235 INFO:tasks.workunit.client.0.smithi039.stderr:194706: 2024-06-03T06:37:10.801+0000 7f2680419640 -1 osd.1 70 log_to_monitors true 2024-06-03T06:37:12.235 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:12.235 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:12.235 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:12.235 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:37:12.235 INFO:tasks.workunit.client.0.smithi039.stderr:194706: 3 2024-06-03T06:37:12.235 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:12.235 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:37:12.239 INFO:tasks.workunit.client.0.smithi039.stderr:194706: osd.1 up in weight 1 up_from 76 up_thru 69 down_aovery_ops 2024-06-03T06:37:12.239 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-03T06:37:12.239 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-03T06:37:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:37:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:37:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:37:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:194708: 2024-06-03T06:37:07.700+0000 7f4ed6168640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:194708: 2024-06-03T06:37:07.702+0000 7f4ed6168640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:194708: 2024-06-03T06:37:07.703+0000 7f4ed6168640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-03T06:37:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:37:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:37:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:37:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:37:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:37:12.241 INFO:tasks.workunit.client.0.smithi039.stderr:194708: 0 2024-06-03T06:37:12.241 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:12.241 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:37:12.241 INFO:tasks.workunit.client.0.smithi039.stderr:194708: 2024-06-03T06:37:08.221+0000 7f4ed6168640 -1 Falling back to public interface 2024-06-03T06:37:12.241 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:12.241 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:12.241 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:12.241 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:37:12.241 INFO:tasks.workunit.client.0.smithi039.stderr:194708: 1 2024-06-03T06:37:12.241 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:12.241 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:37:12.241 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:12.242 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:12.242 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:12.242 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:37:12.242 INFO:tasks.workunit.client.0.smithi039.stderr:194708: 2 2024-06-03T06:37:12.242 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:12.242 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:37:12.242 INFO:tasks.workunit.client.0.smithi039.stderr:194708: 2024-06-03T06:37:10.870+0000 7f4ed6168640 -1 osd.2 70 log_to_monitors true 2024-06-03T06:37:12.242 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:12.242 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:12.242 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:12.242 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:37:12.243 INFO:tasks.workunit.client.0.smithi039.stderr:194708: 3 2024-06-03T06:37:12.243 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:12.243 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:37:13.310 INFO:tasks.workunit.client.0.smithi039.stderr:194708: osd.2 up in weight 1 up_from 77 up_thru 50 down_at 71 last_clean_interval [67,70) [v2:127.0.0.1:6810/3651689020,v1:127.0.0.1:6811/3651689020] [v2:127.0.0.1:6812/3651689020,v1:127.0.0.1:6813/3651689020] exists,up c1bf0b94-99bf-4fb0-823c-5047aa4d2dd5 2024-06-03T06:37:13.311 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:37:13.311 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:37:13.311 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:37:13.311 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:37:13.311 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:37:13.311 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:37:13.311 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:37:13.311 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:37:13.311 INFO:tasks.workunit.client.0.smithi039.stderr:194706: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:37:13.311 INFO:tasks.workunit.client.0.smithi039.stderr:194706: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:37:13.312 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:37:13.312 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:37:13.312 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:37:13.312 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:37:13.312 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:37:13.312 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:37:13.312 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:37:13.312 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:37:13.312 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:37:13.312 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:37:13.312 INFO:tasks.workunit.client.0.smithi039.stderr:194706: 1 2024-06-03T06:37:13.312 INFO:tasks.workunit.client.0.smithi039.stderr:194706: 2 2024-06-03T06:37:13.312 INFO:tasks.workunit.client.0.smithi039.stderr:194706: 3' 2024-06-03T06:37:13.313 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:37:13.313 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:13.313 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:37:13.313 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836540 2024-06-03T06:37:13.313 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836540 2024-06-03T06:37:13.313 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540' 2024-06-03T06:37:13.313 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:13.313 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:37:13.313 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514498 2024-06-03T06:37:13.313 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514498 2024-06-03T06:37:13.313 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-326417514498' 2024-06-03T06:37:13.313 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:13.314 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stat 71 last_clean_interval [69,70) [v2:127.0.0.1:6818/1844373294,v1:127.0.0.1:6819/1844373294] [v2:127.0.0.1:6820/1844373294,v1:127.0.0.1:6821/1844373294] exists,up 5cd6cf24-6678-4c6e-9721-cd271fc1f0af 2024-06-03T06:37:13.314 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:37:13.314 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:37:13.314 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:37:13.314 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:37:13.314 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:37:13.314 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:37:13.314 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:37:13.314 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:37:13.314 INFO:tasks.workunit.client.0.smithi039.stderr:194708: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:37:13.314 INFO:tasks.workunit.client.0.smithi039.stderr:194708: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:37:13.314 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:37:13.314 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:37:13.315 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:37:13.315 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:37:13.315 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:37:13.315 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:37:13.315 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:37:13.315 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:37:13.315 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:37:13.315 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:37:13.315 INFO:tasks.workunit.client.0.smithi039.stderr:194708: 1 2024-06-03T06:37:13.315 INFO:tasks.workunit.client.0.smithi039.stderr:194708: 2 2024-06-03T06:37:13.315 INFO:tasks.workunit.client.0.smithi039.stderr:194708: 3' 2024-06-03T06:37:13.315 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:37:13.315 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:13.316 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:37:13.316 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836539 2024-06-03T06:37:13.316 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836539 2024-06-03T06:37:13.316 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539' 2024-06-03T06:37:13.316 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:13.316 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:37:13.316 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514499 2024-06-03T06:37:13.316 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514499 2024-06-03T06:37:13.316 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-326417514499' 2024-06-03T06:37:13.316 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:15.710 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:37:15.711 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-06-03T06:37:15.711 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-06-03T06:37:15.711 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-326417514498 2-330712481794' 2024-06-03T06:37:15.711 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:15.711 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:37:15.711 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037773 2024-06-03T06:37:15.711 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037773 2024-06-03T06:37:15.711 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-326417514498 2-330712481794 3-257698037773' 2024-06-03T06:37:15.711 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:15.711 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836540 2024-06-03T06:37:15.711 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:15.712 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:37:15.712 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836540 2024-06-03T06:37:15.712 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:15.712 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836540 2024-06-03T06:37:15.712 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-06-03T06:37:15.712 INFO:tasks.workunit.client.0.smithi039.stderr:194706: waiting osd.0 seq 21474836540 2024-06-03T06:37:15.712 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:37:15.712 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836540 2024-06-03T06:37:15.712 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:37:15.712 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:37:15.712 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:37:15.712 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836540 -lt 21474836540 2024-06-03T06:37:15.712 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:15.713 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-326417514498 2024-06-03T06:37:15.713 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:15.713 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:37:15.713 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-326417514498 2024-06-03T06:37:15.713 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:15.713 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514498 2024-06-03T06:37:15.713 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 326417514498' 2024-06-03T06:37:15.713 INFO:tasks.workunit.client.0.smithi039.stderr:194706: waiting osd.1 seq 326417514498 2024-06-03T06:37:15.713 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:37:15.713 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514498 2024-06-03T06:37:15.716 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in ts: ceph tell osd.2 flush_pg_stats 2024-06-03T06:37:15.716 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481795 2024-06-03T06:37:15.716 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481795 2024-06-03T06:37:15.716 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-326417514499 2-330712481795' 2024-06-03T06:37:15.716 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:15.716 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:37:15.716 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037772 2024-06-03T06:37:15.716 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037772 2024-06-03T06:37:15.716 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-326417514499 2-330712481795 3-257698037772' 2024-06-03T06:37:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836539 2024-06-03T06:37:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:37:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836539 2024-06-03T06:37:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836539 2024-06-03T06:37:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2024-06-03T06:37:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:194708: waiting osd.0 seq 21474836539 2024-06-03T06:37:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:37:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836539 2024-06-03T06:37:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:37:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:37:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:37:15.718 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836540 -lt 21474836539 2024-06-03T06:37:15.718 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:15.718 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-326417514499 2024-06-03T06:37:15.718 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:15.718 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:37:15.718 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-326417514499 2024-06-03T06:37:15.718 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:15.718 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514499 2024-06-03T06:37:15.718 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 326417514499' 2024-06-03T06:37:15.718 INFO:tasks.workunit.client.0.smithi039.stderr:194708: waiting osd.1 seq 326417514499 2024-06-03T06:37:15.718 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:37:15.718 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514499 2024-06-03T06:37:17.006 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:17.007 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481794 2024-06-03T06:37:17.007 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:17.007 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:37:17.007 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481794 2024-06-03T06:37:17.007 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:17.007 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-06-03T06:37:17.007 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481794' 2024-06-03T06:37:17.007 INFO:tasks.workunit.client.0.smithi039.stderr:194706: waiting osd.2 seq 330712481794 2024-06-03T06:37:17.007 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:37:17.007 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-06-03T06:37:17.007 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037773 2024-06-03T06:37:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:37:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037773 2024-06-03T06:37:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037773 2024-06-03T06:37:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037773' 2024-06-03T06:37:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:194706: waiting osd.3 seq 257698037773 2024-06-03T06:37:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:37:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037773 -lt 257698037773 2024-06-03T06:37:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:37:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:37:17.009 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:37:17.009 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:37:17.009 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:37:17.009 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:37:17.009 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:37:17.009 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:37:17.009 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:37:17.009 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:37:17.009 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:37:17.036 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephte$seqs 2024-06-03T06:37:17.036 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481795 2024-06-03T06:37:17.036 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:37:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481795 2024-06-03T06:37:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481795 2024-06-03T06:37:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481795' 2024-06-03T06:37:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:194708: waiting osd.2 seq 330712481795 2024-06-03T06:37:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:37:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481795 2024-06-03T06:37:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037772 2024-06-03T06:37:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:37:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037772 2024-06-03T06:37:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037772 2024-06-03T06:37:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037772' 2024-06-03T06:37:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:194708: waiting osd.3 seq 257698037772 2024-06-03T06:37:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:37:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037773 -lt 257698037772 2024-06-03T06:37:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:37:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:37:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:37:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:37:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:37:17.039 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:37:17.039 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:37:17.039 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:37:17.039 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:37:17.039 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:37:17.039 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:37:17.372 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:17.373 INFO:tasks.workunit.client.0.smithi039.stderr:194706: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:17.373 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:37:17.373 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:37:17.373 INFO:tasks.workunit.client.0.smithi039.stderr:194706: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:37:17.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:37:17.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:37:17.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 194705 2024-06-03T06:37:17.401 INFO:tasks.workunit.client.0.smithi039.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:17.401 INFO:tasks.workunit.client.0.smithi039.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:17.401 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:37:17.401 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:37:17.401 INFO:tasks.workunit.client.0.smithi039.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:37:17.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:37:17.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:37:17.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:37:17.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:37:17.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-03T06:37:17.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:37:17.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:37:17.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:37:17.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-06-03T06:37:17.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-03T06:37:17.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-03T06:37:17.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-06-03T06:37:17.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-06-03T06:37:17.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-03T06:37:17.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:37:17.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:37:17.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:37:17.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 197382"' 2024-06-03T06:37:17.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 197382' 2024-06-03T06:37:17.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:37:17.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:37:17.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:37:17.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 197384"' 2024-06-03T06:37:17.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:37:17.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 197384' 2024-06-03T06:37:17.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-03T06:37:17.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 197382 197384' 2024-06-03T06:37:17.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:37:17.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:37:17.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 197382 2024-06-03T06:37:17.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/197385: /' 2024-06-03T06:37:17.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:37:17.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/197388: /' 2024-06-03T06:37:20.072 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:37:20.073 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:37:20.073 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:37:20.073 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:37:20.073 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:37:20.073 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:37:20.073 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:37:20.073 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:37:20.073 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:37:20.073 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:37:20.073 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:37:20.073 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:37:20.073 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:37:20.073 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:37:20.073 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:37:20.074 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:37:20.074 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:37:20.074 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:37:20.074 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:37:20.074 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:37:20.074 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:37:20.074 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:37:20.074 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-03T06:37:20.074 INFO:tasks.workunit.client.0.smithi039.stderr:197388: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-03T06:37:20.074 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:37:20.074 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:37:20.074 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:37:20.074 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:37:20.074 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:37:20.075 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:37:20.075 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:37:20.075 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:37:20.076 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:37:20.076 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:37:20.076 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:37:20.076 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:37:20.076 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:37:20.076 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:37:20.076 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:37:20.076 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:37:20.076 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:37:20.077 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:37:20.077 INFO:tasks.workunit.client.0.smithi039.stderr:197388: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:37:20.077 INFO:tasks.workunit.client.0.smithi039.stderr:197388: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:37:20.077 INFO:tasks.workunit.client.0.smithi039.stderr:197388: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:37:20.077 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:20.077 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:20.077 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:37:20.077 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:37:20.077 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:37:20.077 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:37:20.077 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:37:20.077 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:37:20.077 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:37:20.077 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:37:20.078 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:37:20.078 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:37:20.078 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:37:20.078 INFO:tasks.workunit.client.0.smithi039.stderr:197388: start osd.1 2024-06-03T06:37:20.078 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:37:20.556 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:37:20.556 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:37:20.557 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:37:20.557 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:37:20.557 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:37:20.557 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:37:20.557 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:37:20.557 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:37:20.557 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:37:20.557 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:37:20.557 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:37:20.557 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:37:20.557 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:37:20.557 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:37:20.557 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:37:20.558 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:37:20.558 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:37:20.558 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:37:20.558 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:37:20.558 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:37:20.558 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:37:20.558 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:37:20.558 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-03T06:37:20.558 INFO:tasks.workunit.client.0.smithi039.stderr:197385: remove 0#2:eb822e21:::SOMETHING:head# 2024-06-03T06:37:20.558 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:37:20.558 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:37:20.558 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:37:20.559 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:37:20.559 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:37:20.559 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:37:20.559 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:37:20.559 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:37:20.559 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:37:20.559 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:37:20.559 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:37:20.559 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:37:20.559 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:37:20.560 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:37:20.560 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:37:20.560 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:37:20.560 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:37:20.560 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:37:20.560 INFO:tasks.workunit.client.0.smithi039.stderr:197385: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:37:20.560 INFO:tasks.workunit.client.0.smithi039.stderr:197385: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:37:20.560 INFO:tasks.workunit.client.0.smithi039.stderr:197385: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:37:20.560 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:20.560 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:20.560 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:37:20.560 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:37:20.560 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:37:20.561 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:37:20.561 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:37:20.561 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:37:20.561 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:37:20.561 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:37:20.561 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:37:20.561 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:37:20.561 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:37:20.561 INFO:tasks.workunit.client.0.smithi039.stderr:197385: start osd.3 2024-06-03T06:37:20.561 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:37:24.666 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:37:24.666 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:37:24.666 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:37:24.666 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:37:24.666 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:37:24.667 INFO:tasks.workunit.client.0.smithi039.stderr:197388: 2024-06-03T06:37:20.106+0000 7f8ee500e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:24.667 INFO:tasks.workunit.client.0.smithi039.stderr:197388: 2024-06-03T06:37:20.109+0000 7f8ee500e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:24.667 INFO:tasks.workunit.client.0.smithi039.stderr:197388: 2024-06-03T06:37:20.110+0000 7f8ee500e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:24.667 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:37:24.667 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:37:24.667 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:37:24.667 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:37:24.667 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:37:24.667 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:24.667 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:37:24.667 INFO:tasks.workunit.client.0.smithi039.stderr:197388: 0 2024-06-03T06:37:24.667 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:24.667 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:37:24.668 INFO:tasks.workunit.client.0.smithi039.stderr:197388: 2024-06-03T06:37:20.633+0000 7f8ee500e640 -1 Falling back to public interface 2024-06-03T06:37:24.668 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:24.668 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:24.668 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:24.668 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:37:24.668 INFO:tasks.workunit.client.0.smithi039.stderr:197388: 1 2024-06-03T06:37:24.668 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:24.668 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:37:24.668 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:24.668 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:24.668 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:24.668 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:37:24.668 INFO:tasks.workunit.client.0.smithi039.stderr:197388: 2 2024-06-03T06:37:24.669 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:24.669 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:37:24.669 INFO:tasks.workunit.client.0.smithi039.stderr:197388: 2024-06-03T06:37:23.234+0000 7f8ee500e640 -1 osd.1 79 log_to_monitors true 2024-06-03T06:37:24.669 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:24.669 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:24.669 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:24.669 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:37:24.669 INFO:tasks.workunit.client.0.smithi039.stderr:197388: 3 2024-06-03T06:37:24.669 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:24.669 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:37:25.308 INFO:tasks.workunit.client.0.smithi039.stderr:197388: osd.1 up in weight 1 up_from 83 up_thru 78 down_at 80 last_197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:37:25.308 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:37:25.309 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:37:25.309 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:37:25.309 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:37:25.309 INFO:tasks.workunit.client.0.smithi039.stderr:197385: 2024-06-03T06:37:20.586+0000 7f77853dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:25.309 INFO:tasks.workunit.client.0.smithi039.stderr:197385: 2024-06-03T06:37:20.589+0000 7f77853dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:25.309 INFO:tasks.workunit.client.0.smithi039.stderr:197385: 2024-06-03T06:37:20.590+0000 7f77853dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:25.309 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:37:25.309 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:37:25.309 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:37:25.309 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:37:25.309 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:37:25.309 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:25.310 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:37:25.310 INFO:tasks.workunit.client.0.smithi039.stderr:197385: 0 2024-06-03T06:37:25.310 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:25.310 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:37:25.310 INFO:tasks.workunit.client.0.smithi039.stderr:197385: 2024-06-03T06:37:21.106+0000 7f77853dc640 -1 Falling back to public interface 2024-06-03T06:37:25.310 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:25.310 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:25.310 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:25.310 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:37:25.310 INFO:tasks.workunit.client.0.smithi039.stderr:197385: 1 2024-06-03T06:37:25.310 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:25.310 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:37:25.310 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:25.311 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:25.311 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:25.311 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:37:25.311 INFO:tasks.workunit.client.0.smithi039.stderr:197385: 2 2024-06-03T06:37:25.311 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:25.311 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:37:25.311 INFO:tasks.workunit.client.0.smithi039.stderr:197385: 2024-06-03T06:37:23.806+0000 7f77853dc640 -1 osd.3 79 log_to_monitors true 2024-06-03T06:37:25.311 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:25.311 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:25.311 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:25.311 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:37:25.311 INFO:tasks.workunit.client.0.smithi039.stderr:197385: 3 2024-06-03T06:37:25.311 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:25.311 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:37:25.688 INFO:tasks.workunit.client.0.smithi039.stderr:197385: osd.3 up in weight 1 up_from 85 up_thru 85 down_at 80 last_clean_interval [76,79) [v2:127.0.0.1:6810/3617344703,v1:127.0.0.1:6811/3617344703] [v2:127.0.0.1:6812/3617344703,v1:127.0.0.1:6813/3617344703] exists,up c1bf0b94-99bf-4fb0-823c-5047aa4d2dd5 2024-06-03T06:37:25.688 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:37:25.688 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:37:25.688 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:37:25.688 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:37:25.688 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:37:25.688 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:37:25.688 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:37:25.689 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:37:25.689 INFO:tasks.workunit.client.0.smithi039.stderr:197388: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:37:25.689 INFO:tasks.workunit.client.0.smithi039.stderr:197388: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:37:25.689 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:37:25.689 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:37:25.689 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:37:25.689 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:37:25.689 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:37:25.689 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:37:25.689 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:37:25.689 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:37:25.689 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:37:25.689 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:37:25.690 INFO:tasks.workunit.client.0.smithi039.stderr:197388: 1 2024-06-03T06:37:25.690 INFO:tasks.workunit.client.0.smithi039.stderr:197388: 2 2024-06-03T06:37:25.690 INFO:tasks.workunit.client.0.smithi039.stderr:197388: 3' 2024-06-03T06:37:25.690 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:37:25.690 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:25.690 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:37:25.690 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836543 2024-06-03T06:37:25.690 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836543 2024-06-03T06:37:25.690 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543' 2024-06-03T06:37:25.690 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:25.690 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:37:25.690 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285570 2024-06-03T06:37:25.691 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285570 2024-06-03T06:37:25.691 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-356482285570' 2024-06-03T06:37:25.691 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:26.354 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [60,79) [v2:127.0.0.1:6826/2451772781,v1:127.0.0.1:6827/2451772781] [v2:127.0.0.1:6828/2451772781,v1:127.0.0.1:6829/2451772781] exists,up 6f805989-769d-4078-8009-2c052ac9acc1 2024-06-03T06:37:26.355 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:37:26.355 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:37:26.355 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:37:26.355 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:37:26.355 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:37:26.355 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:37:26.355 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:37:26.355 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:37:26.355 INFO:tasks.workunit.client.0.smithi039.stderr:197385: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:37:26.355 INFO:tasks.workunit.client.0.smithi039.stderr:197385: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:37:26.355 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:37:26.356 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:37:26.356 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:37:26.356 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:37:26.356 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:37:26.356 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:37:26.356 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:37:26.356 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:37:26.356 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:37:26.356 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:37:26.356 INFO:tasks.workunit.client.0.smithi039.stderr:197385: 1 2024-06-03T06:37:26.356 INFO:tasks.workunit.client.0.smithi039.stderr:197385: 2 2024-06-03T06:37:26.356 INFO:tasks.workunit.client.0.smithi039.stderr:197385: 3' 2024-06-03T06:37:26.356 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:37:26.356 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:26.356 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:37:26.357 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836544 2024-06-03T06:37:26.357 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836544 2024-06-03T06:37:26.357 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544' 2024-06-03T06:37:26.357 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:26.357 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:37:26.357 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285571 2024-06-03T06:37:26.357 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285571 2024-06-03T06:37:26.357 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-356482285571' 2024-06-03T06:37:26.357 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:28.127 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:37:28.127 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481798 2024-06-03T06:37:28.127 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481798 2024-06-03T06:37:28.127 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-356482285570 2-330712481798' 2024-06-03T06:37:28.127 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:28.127 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:37:28.127 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220162 2024-06-03T06:37:28.127 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220162 2024-06-03T06:37:28.127 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-356482285570 2-330712481798 3-365072220162' 2024-06-03T06:37:28.127 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:28.127 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836543 2024-06-03T06:37:28.127 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:28.128 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:37:28.128 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836543 2024-06-03T06:37:28.128 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:28.128 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836543 2024-06-03T06:37:28.128 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-06-03T06:37:28.128 INFO:tasks.workunit.client.0.smithi039.stderr:197388: waiting osd.0 seq 21474836543 2024-06-03T06:37:28.128 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:37:28.128 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836543 2024-06-03T06:37:28.128 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:37:28.128 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:37:28.128 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:37:28.128 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836545 -lt 21474836543 2024-06-03T06:37:28.128 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:28.128 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-356482285570 2024-06-03T06:37:28.129 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:28.129 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:37:28.129 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-356482285570 2024-06-03T06:37:28.129 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:28.129 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285570 2024-06-03T06:37:28.129 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 356482285570' 2024-06-03T06:37:28.129 INFO:tasks.workunit.client.0.smithi039.stderr:197388: waiting osd.1 seq 356482285570 2024-06-03T06:37:28.129 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:37:28.129 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285571 -lt 356482285570 2024-06-03T06:37:28.129 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:28.481 INFO:tasks.workunit.client.0.smithi039.stderr:1973tell osd.2 flush_pg_stats 2024-06-03T06:37:28.481 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481799 2024-06-03T06:37:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481799 2024-06-03T06:37:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-356482285571 2-330712481799' 2024-06-03T06:37:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:37:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220163 2024-06-03T06:37:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220163 2024-06-03T06:37:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-356482285571 2-330712481799 3-365072220163' 2024-06-03T06:37:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836544 2024-06-03T06:37:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:37:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836544 2024-06-03T06:37:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836544 2024-06-03T06:37:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-06-03T06:37:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:197385: waiting osd.0 seq 21474836544 2024-06-03T06:37:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:37:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836544 2024-06-03T06:37:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:37:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:37:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:37:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836545 -lt 21474836544 2024-06-03T06:37:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-356482285571 2024-06-03T06:37:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:28.484 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:37:28.484 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-356482285571 2024-06-03T06:37:28.484 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:28.484 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285571 2024-06-03T06:37:28.484 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 356482285571' 2024-06-03T06:37:28.484 INFO:tasks.workunit.client.0.smithi039.stderr:197385: waiting osd.1 seq 356482285571 2024-06-03T06:37:28.484 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:37:28.484 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285571 -lt 356482285571 2024-06-03T06:37:28.484 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:29.469 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481798 2024-06-03T06:37:29.470 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:29.470 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:37:29.470 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481798 2024-06-03T06:37:29.470 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:29.470 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481798 2024-06-03T06:37:29.470 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481798' 2024-06-03T06:37:29.470 INFO:tasks.workunit.client.0.smithi039.stderr:197388: waiting osd.2 seq 330712481798 2024-06-03T06:37:29.470 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:37:29.470 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481800 -lt 330712481798 2024-06-03T06:37:29.470 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:29.470 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-365072220162 2024-06-03T06:37:29.470 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:29.470 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:37:29.472 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-365072220162 2024-06-03T06:37:29.472 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:29.472 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220162 2024-06-03T06:37:29.472 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 365072220162' 2024-06-03T06:37:29.473 INFO:tasks.workunit.client.0.smithi039.stderr:197388: waiting osd.3 seq 365072220162 2024-06-03T06:37:29.473 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:37:29.473 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220163 -lt 365072220162 2024-06-03T06:37:29.473 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:37:29.473 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:29.473 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:29.473 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:37:29.474 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:37:29.474 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:37:29.474 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:37:29.474 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:37:29.474 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:37:29.474 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:37:29.475 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:37:29.475 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:37:29.475 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:37:29.838 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:29.838 INFO:tasks.workunit.client.0.smithi039.stderr:197388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:29.838 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:37:29.839 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:37:29.839 INFO:tasks.workunit.client.0.smithi039.stderr:197388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:37:29.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:37:29.879 INFO:tasks.workunit.client.0.smithi039.stderr:85: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481799 2024-06-03T06:37:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:37:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481799 2024-06-03T06:37:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481799 2024-06-03T06:37:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481799' 2024-06-03T06:37:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:197385: waiting osd.2 seq 330712481799 2024-06-03T06:37:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:37:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481800 -lt 330712481799 2024-06-03T06:37:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-365072220163 2024-06-03T06:37:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:37:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-365072220163 2024-06-03T06:37:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220163 2024-06-03T06:37:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 365072220163' 2024-06-03T06:37:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:197385: waiting osd.3 seq 365072220163 2024-06-03T06:37:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:37:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220163 -lt 365072220163 2024-06-03T06:37:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:37:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:37:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:37:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:37:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:37:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:37:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:37:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:37:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:37:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:37:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:37:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:197385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:37:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:37:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:197385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:37:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:37:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:37:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 197384 2024-06-03T06:37:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:37:30.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:37:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:37:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-03T06:37:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:37:30.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-03T06:37:30.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:37:30.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:37:30.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:37:30.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:37:30.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-03T06:37:30.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-03T06:37:30.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:37:30.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:37:30.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:37:30.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:37:30.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:37:30.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:37:30.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:37:00.485422+0000 2024-06-03T06:37:30.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:37:31.119 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:37:31.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:37:00.485422+0000 2024-06-03T06:37:31.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:37:31.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:37:00.485422+0000 2024-06-03T06:37:31.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:37:31.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:37:31.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:37:31.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:37:31.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:37:31.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:37:31.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:37:31.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:37:31.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:37:00.485422+0000 '>' 2024-06-03T06:37:00.485422+0000 2024-06-03T06:37:31.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:37:32.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:37:32.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:37:32.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:37:32.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:37:32.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:37:32.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:37:32.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:37:32.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:37:00.485422+0000 '>' 2024-06-03T06:37:00.485422+0000 2024-06-03T06:37:32.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:37:33.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:37:33.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:37:33.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:37:33.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:37:33.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:37:33.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:37:33.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:37:34.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:37:00.485422+0000 '>' 2024-06-03T06:37:00.485422+0000 2024-06-03T06:37:34.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:37:35.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:37:35.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:37:35.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:37:35.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:37:35.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:37:35.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:37:35.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:37:35.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:37:31.683548+0000 '>' 2024-06-03T06:37:00.485422+0000 2024-06-03T06:37:35.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:37:35.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-03T06:37:35.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:37:35.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:37:35.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:37:35.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 200209"' 2024-06-03T06:37:35.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 200209' 2024-06-03T06:37:35.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:37:35.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:37:35.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:37:35.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 200211"' 2024-06-03T06:37:35.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 200211' 2024-06-03T06:37:35.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:37:35.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-03T06:37:35.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 200209 200211' 2024-06-03T06:37:35.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:37:35.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:37:35.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 200209 2024-06-03T06:37:35.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/200212: /' 2024-06-03T06:37:35.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:37:35.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/200214: /' 2024-06-03T06:37:37.865 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:37:37.865 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:37:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:37:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:37:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:37:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:37:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:37:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:37:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:37:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:37:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:37:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:37:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:37:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:37:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:37:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:37:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:37:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:37:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:37:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:37:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:37:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:37:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-03T06:37:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:200212: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-03T06:37:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:200212: _ 2024-06-03T06:37:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:200212: hinfo_key 2024-06-03T06:37:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:200212: snapset 2024-06-03T06:37:37.867 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:37:37.868 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:37:37.868 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:37:37.868 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:37:37.868 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:37:37.868 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:37:37.868 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:37:37.869 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:37:37.869 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:37:37.869 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:37:37.869 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:37:37.869 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:37:37.869 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:37:37.869 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:37:37.869 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:37:37.869 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:37:37.869 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:37:37.870 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:37:37.870 INFO:tasks.workunit.client.0.smithi039.stderr:200212: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:37:37.870 INFO:tasks.workunit.client.0.smithi039.stderr:200212: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:37:37.870 INFO:tasks.workunit.client.0.smithi039.stderr:200212: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:37:37.870 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:37.870 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:37.870 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:37:37.870 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:37:37.870 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:37:37.870 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:37:37.870 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:37:37.870 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:37:37.870 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:37:37.871 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:37:37.871 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:37:37.871 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:37:37.871 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:37:37.871 INFO:tasks.workunit.client.0.smithi039.stderr:200212: start osd.3 2024-06-03T06:37:38.515 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:37:38.515 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:37:38.516 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:37:38.516 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:37:38.516 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:37:38.516 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:37:38.516 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:37:38.516 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:37:38.517 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:37:38.517 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:37:38.517 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:37:38.517 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:37:38.517 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:37:38.517 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:37:38.517 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:37:38.517 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:37:38.517 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:37:38.517 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:37:38.517 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:37:38.517 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:37:38.518 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:37:38.518 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:37:38.518 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-03T06:37:38.518 INFO:tasks.workunit.client.0.smithi039.stderr:200214: _ 2024-06-03T06:37:38.518 INFO:tasks.workunit.client.0.smithi039.stderr:200214: hinfo_key 2024-06-03T06:37:38.518 INFO:tasks.workunit.client.0.smithi039.stderr:200214: snapset 2024-06-03T06:37:38.518 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:37:38.518 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:37:38.518 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:37:38.519 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:37:38.519 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:37:38.519 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:37:38.519 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:37:38.519 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:37:38.519 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:37:38.519 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:37:38.519 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:37:38.520 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:37:38.520 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:37:38.520 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:37:38.520 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:37:38.520 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:37:38.520 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:37:38.520 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:37:38.520 INFO:tasks.workunit.client.0.smithi039.stderr:200214: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:37:38.521 INFO:tasks.workunit.client.0.smithi039.stderr:200214: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:37:38.521 INFO:tasks.workunit.client.0.smithi039.stderr:200214: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:37:38.521 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:38.521 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:38.521 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:37:38.522 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:37:38.522 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:37:38.522 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:37:38.522 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:37:38.522 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:37:38.522 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:37:38.522 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:37:38.522 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:37:38.523 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:37:38.523 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:37:38.523 INFO:tasks.workunit.client.0.smithi039.stderr:200214: start osd.1 2024-06-03T06:37:42.076 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:37:42.077 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:37:42.077 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:37:42.077 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:37:42.077 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:37:42.077 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:37:42.077 INFO:tasks.workunit.client.0.smithi039.stderr:200212: 2024-06-03T06:37:37.899+0000 7fd0acfdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:42.077 INFO:tasks.workunit.client.0.smithi039.stderr:200212: 2024-06-03T06:37:37.950+0000 7fd0acfdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:42.077 INFO:tasks.workunit.client.0.smithi039.stderr:200212: 2024-06-03T06:37:37.951+0000 7fd0acfdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:42.077 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:37:42.077 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:37:42.077 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:37:42.077 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:37:42.078 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:37:42.078 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:42.078 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:37:42.078 INFO:tasks.workunit.client.0.smithi039.stderr:200212: 0 2024-06-03T06:37:42.078 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:42.078 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:37:42.078 INFO:tasks.workunit.client.0.smithi039.stderr:200212: 2024-06-03T06:37:38.466+0000 7fd0acfdc640 -1 Falling back to public interface 2024-06-03T06:37:42.078 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:42.078 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:42.078 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:42.078 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:37:42.078 INFO:tasks.workunit.client.0.smithi039.stderr:200212: 1 2024-06-03T06:37:42.078 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:37:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:37:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:200212: 2 2024-06-03T06:37:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:37:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:200212: 2024-06-03T06:37:40.958+0000 7fd0acfdc640 -1 osd.3 87 log_to_monitors true 2024-06-03T06:37:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:37:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:200212: 3 2024-06-03T06:37:42.080 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:43.170 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-06-03T06:37:43.170 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:37:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:37:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:37:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:37:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:37:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:200214: 2024-06-03T06:37:38.544+0000 7f297e423640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:200214: 2024-06-03T06:37:38.551+0000 7f297e423640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:200214: 2024-06-03T06:37:38.553+0000 7f297e423640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:37:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:37:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:37:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:37:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:37:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: 0 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: 2024-06-03T06:37:39.067+0000 7f297e423640 -1 Falling back to public interface 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: 1 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:43.172 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:37:43.173 INFO:tasks.workunit.client.0.smithi039.stderr:200214: 2 2024-06-03T06:37:43.173 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:43.173 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:37:43.173 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:43.173 INFO:tasks.workunit.client.0.smithi039.stderr:200214: 2024-06-03T06:37:41.748+0000 7f297e423640 -1 osd.1 87 log_to_monitors true 2024-06-03T06:37:43.173 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:43.173 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:43.173 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:37:43.173 INFO:tasks.workunit.client.0.smithi039.stderr:200214: 3 2024-06-03T06:37:43.173 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:43.173 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:37:43.428 INFO:tasks.workunit.client.0.smithi039.stderr:200214: osd.1 up in weight 1 up_from 95 up_thru 83 down_as.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:37:43.428 INFO:tasks.workunit.client.0.smithi039.stderr:200212: osd.3 up in weight 1 up_from 92 up_thru 85 down_at 88 last_clean_interval [85,87) [v2:127.0.0.1:6810/3160770137,v1:127.0.0.1:6811/3160770137] [v2:127.0.0.1:6812/3160770137,v1:127.0.0.1:6813/3160770137] exists,up 6f805989-769d-4078-8009-2c052ac9acc1 2024-06-03T06:37:43.428 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:37:43.428 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:37:43.428 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:37:43.428 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:37:43.428 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:37:43.428 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:37:43.428 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:37:43.428 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:37:43.429 INFO:tasks.workunit.client.0.smithi039.stderr:200212: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:37:43.429 INFO:tasks.workunit.client.0.smithi039.stderr:200212: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:37:43.429 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:37:43.429 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:37:43.429 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:37:43.429 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:37:43.429 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:37:43.429 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:37:43.429 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:37:43.429 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:37:43.429 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:37:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:37:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:200212: 1 2024-06-03T06:37:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:200212: 2 2024-06-03T06:37:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:200212: 3' 2024-06-03T06:37:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:37:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:37:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836549 2024-06-03T06:37:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836549 2024-06-03T06:37:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549' 2024-06-03T06:37:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:37:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893122 2024-06-03T06:37:43.431 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893122 2024-06-03T06:37:43.431 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-408021893122' 2024-06-03T06:37:44.081 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd it 88 last_clean_interval [83,87) [v2:127.0.0.1:6826/2393469065,v1:127.0.0.1:6827/2393469065] [v2:127.0.0.1:6828/2393469065,v1:127.0.0.1:6829/2393469065] exists,up c1bf0b94-99bf-4fb0-823c-5047aa4d2dd5 2024-06-03T06:37:44.081 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:37:44.081 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:37:44.081 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:37:44.082 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:37:44.082 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:37:44.082 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:37:44.082 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:37:44.082 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:37:44.082 INFO:tasks.workunit.client.0.smithi039.stderr:200214: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:37:44.082 INFO:tasks.workunit.client.0.smithi039.stderr:200214: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:37:44.082 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:37:44.082 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:37:44.082 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:37:44.082 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:37:44.082 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:37:44.083 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:37:44.083 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:37:44.083 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:37:44.083 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:37:44.083 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:37:44.083 INFO:tasks.workunit.client.0.smithi039.stderr:200214: 1 2024-06-03T06:37:44.083 INFO:tasks.workunit.client.0.smithi039.stderr:200214: 2 2024-06-03T06:37:44.083 INFO:tasks.workunit.client.0.smithi039.stderr:200214: 3' 2024-06-03T06:37:44.083 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:37:44.083 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:44.083 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:37:44.083 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836550 2024-06-03T06:37:44.083 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836550 2024-06-03T06:37:44.083 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550' 2024-06-03T06:37:44.084 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:44.084 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:37:44.084 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893123 2024-06-03T06:37:44.084 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893123 2024-06-03T06:37:44.084 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-408021893123' 2024-06-03T06:37:44.084 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:45.853 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stan $ids 2024-06-03T06:37:45.853 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:37:45.853 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481804 2024-06-03T06:37:45.853 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481804 2024-06-03T06:37:45.853 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-408021893122 2-330712481804' 2024-06-03T06:37:45.854 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:45.854 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:37:45.854 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991234 2024-06-03T06:37:45.854 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991234 2024-06-03T06:37:45.854 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-408021893122 2-330712481804 3-395136991234' 2024-06-03T06:37:45.854 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:45.854 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836549 2024-06-03T06:37:45.854 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:45.854 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:37:45.854 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836549 2024-06-03T06:37:45.854 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:45.854 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836549 2024-06-03T06:37:45.855 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836549' 2024-06-03T06:37:45.855 INFO:tasks.workunit.client.0.smithi039.stderr:200212: waiting osd.0 seq 21474836549 2024-06-03T06:37:45.855 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:37:45.855 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836548 -lt 21474836549 2024-06-03T06:37:45.855 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:37:45.855 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:37:45.855 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:37:45.855 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836549 2024-06-03T06:37:45.855 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:45.855 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-408021893122 2024-06-03T06:37:45.855 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:45.855 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:37:45.855 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-408021893122 2024-06-03T06:37:45.856 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:45.856 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893122 2024-06-03T06:37:45.856 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 408021893122' 2024-06-03T06:37:45.856 INFO:tasks.workunit.client.0.smithi039.stderr:200212: waiting osd.1 seq 408021893122 2024-06-03T06:37:45.856 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:37:45.856 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893123 -lt 408021893122 2024-06-03T06:37:46.474 INFO:tasks.workunit.client.0.smithi039.stderr:200212ts: ceph tell osd.2 flush_pg_stats 2024-06-03T06:37:46.474 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481805 2024-06-03T06:37:46.474 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481805 2024-06-03T06:37:46.474 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-408021893123 2-330712481805' 2024-06-03T06:37:46.474 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:37:46.474 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:37:46.474 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991235 2024-06-03T06:37:46.474 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991235 2024-06-03T06:37:46.474 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-408021893123 2-330712481805 3-395136991235' 2024-06-03T06:37:46.475 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:46.475 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836550 2024-06-03T06:37:46.475 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:46.475 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:37:46.475 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836550 2024-06-03T06:37:46.475 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:46.475 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836550 2024-06-03T06:37:46.475 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-06-03T06:37:46.475 INFO:tasks.workunit.client.0.smithi039.stderr:200214: waiting osd.0 seq 21474836550 2024-06-03T06:37:46.475 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:37:46.475 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836548 -lt 21474836550 2024-06-03T06:37:46.475 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:37:46.475 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:37:46.476 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:37:46.476 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836550 2024-06-03T06:37:46.476 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:46.476 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-408021893123 2024-06-03T06:37:46.476 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:46.476 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:37:46.476 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-408021893123 2024-06-03T06:37:46.476 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:46.476 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893123 2024-06-03T06:37:46.476 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 408021893123' 2024-06-03T06:37:46.476 INFO:tasks.workunit.client.0.smithi039.stderr:200214: waiting osd.1 seq 408021893123 2024-06-03T06:37:46.476 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:37:46.476 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893123 -lt 408021893123 2024-06-03T06:37:47.175 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in : /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:47.175 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481804 2024-06-03T06:37:47.175 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:47.175 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:37:47.176 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481804 2024-06-03T06:37:47.176 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:47.176 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481804 2024-06-03T06:37:47.176 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481804' 2024-06-03T06:37:47.176 INFO:tasks.workunit.client.0.smithi039.stderr:200212: waiting osd.2 seq 330712481804 2024-06-03T06:37:47.176 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:37:47.176 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481805 -lt 330712481804 2024-06-03T06:37:47.176 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:47.176 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-395136991234 2024-06-03T06:37:47.176 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:47.176 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:37:47.176 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-395136991234 2024-06-03T06:37:47.176 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:47.176 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991234 2024-06-03T06:37:47.176 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 395136991234' 2024-06-03T06:37:47.177 INFO:tasks.workunit.client.0.smithi039.stderr:200212: waiting osd.3 seq 395136991234 2024-06-03T06:37:47.177 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:37:47.177 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991235 -lt 395136991234 2024-06-03T06:37:47.177 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:37:47.177 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:47.177 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:47.177 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:37:47.177 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:37:47.177 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:37:47.177 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:37:47.177 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:37:47.177 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:37:47.177 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:37:47.177 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:37:47.177 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T06:37:47.767 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.c$seqs 2024-06-03T06:37:47.767 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481805 2024-06-03T06:37:47.767 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:47.767 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:37:47.767 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481805 2024-06-03T06:37:47.767 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481805 2024-06-03T06:37:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481805' 2024-06-03T06:37:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:200214: waiting osd.2 seq 330712481805 2024-06-03T06:37:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:37:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481805 -lt 330712481805 2024-06-03T06:37:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:37:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-395136991235 2024-06-03T06:37:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:37:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:37:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-395136991235 2024-06-03T06:37:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:37:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991235 2024-06-03T06:37:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 395136991235' 2024-06-03T06:37:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:200214: waiting osd.3 seq 395136991235 2024-06-03T06:37:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:37:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991236 -lt 395136991235 2024-06-03T06:37:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:37:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:37:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:37:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:37:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:37:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:37:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:37:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:37:47.770 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:37:47.770 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T06:37:47.770 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:37:48.768 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtelient.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:37:48.768 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:48.768 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:48.768 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-03T06:37:48.768 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-03T06:37:48.768 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:37:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-03T06:37:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:37:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:37:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:37:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:37:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:37:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:37:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:37:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:37:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:37:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:37:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T06:37:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:37:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-03T06:37:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T06:37:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:37:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:37:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:37:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:37:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:37:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:37:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:37:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-06-03T06:37:49.301 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalonst/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:49.301 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:49.301 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-03T06:37:49.301 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-03T06:37:49.301 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:37:49.301 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-03T06:37:49.302 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:37:49.302 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:37:49.302 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:37:49.302 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:37:49.302 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:37:49.302 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:37:49.302 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:37:49.302 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:37:49.302 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:37:49.302 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:37:49.302 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T06:37:49.302 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:37:49.302 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:49.302 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-03T06:37:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T06:37:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:37:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:37:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:37:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:37:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:37:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:37:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:37:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2024-06-03T06:37:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2024-06-03T06:37:49.586 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephte/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-06-03T06:37:49.586 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-06-03T06:37:49.586 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:37:49.586 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:37:49.586 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:37:49.586 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:37:49.586 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:37:49.587 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:37:49.587 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:37:49.587 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:37:49.587 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:37:49.587 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:37:49.587 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:37:49.587 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:37:49.587 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:49.587 INFO:tasks.workunit.client.0.smithi039.stderr:200212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:49.587 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:37:49.587 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:37:49.587 INFO:tasks.workunit.client.0.smithi039.stderr:200212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:37:49.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:37:49.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:37:49.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 200211 2024-06-03T06:37:50.142 INFO:tasks.workunit.client.0.smithi039.stderr:est/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-06-03T06:37:50.143 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:37:50.143 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:37:50.143 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:37:50.143 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:37:50.143 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:37:50.143 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:37:50.143 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:37:50.143 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:37:50.143 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:37:50.143 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:37:50.143 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:37:50.143 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:37:50.144 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:37:50.144 INFO:tasks.workunit.client.0.smithi039.stderr:200214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:37:50.144 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:37:50.144 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:37:50.144 INFO:tasks.workunit.client.0.smithi039.stderr:200214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:37:50.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:37:50.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:37:50.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:37:50.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:37:50.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-03T06:37:50.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:37:50.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:37:50.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:37:50.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T06:37:50.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:37:50.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:37:50.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:37:50.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:37:50.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:37:50.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:37:50.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:37:50.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:37:50.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:37:50.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:37:50.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:37:50.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:37:50.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:37:50.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:37:50.307 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:37:50.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:37:50.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:37:50.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:37:50.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:37:50.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:37:50.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:37:50.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:37:50.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:37:50.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:37:50.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:37:50.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:37:50.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:37:50.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:37:50.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T06:37:50.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T06:37:50.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T06:37:50.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T06:37:50.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:37:50.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:37:50.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:37:50.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:37:50.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:37:50.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:37:50.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:37:50.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:37:50.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:37:50.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:37:50.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:37:50.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:37:50.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:37:50.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:37:50.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:37:50.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:37:50.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:37:50.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:37:50.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:37:50.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:37:50.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:37:50.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:37:50.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:37:50.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:37:50.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:37:50.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:37:50.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:37:50.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:37:50.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T06:37:50.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:37:50.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:37:50.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:37:50.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T06:37:50.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:37:50.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T06:37:50.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:37:50.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T06:37:50.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-06-03T06:37:50.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-06-03T06:37:50.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-06-03T06:37:50.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-06-03T06:37:50.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-06-03T06:37:50.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-06-03T06:37:50.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T06:37:50.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T06:37:50.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T06:37:50.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T06:37:50.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T06:37:50.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T06:37:50.713 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T06:37:50.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:37:50.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:37:50.714 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:37:50.714 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:37:50.714 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:37:50.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:50.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T06:37:50.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T06:37:50.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T06:37:50.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:37:50.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:37:50.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T06:37:50.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T06:37:50.907 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:37:50.907 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:37:50.907 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:37:50.908 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:37:50.908 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:37:50.908 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:37:50.908 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:37:50.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:37:50.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T06:37:50.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T06:37:50.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:37:50.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:37:50.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T06:37:50.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T06:37:50.991 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:37:50.991 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:37:50.992 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:37:50.992 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:37:50.992 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:37:50.992 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:37:50.992 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:37:50.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:37:50.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T06:37:51.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-06-03T06:37:51.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T06:37:51.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T06:37:51.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T06:37:51.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T06:37:51.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T06:37:51.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T06:37:51.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T06:37:51.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:37:51.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:37:51.891 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:37:51.891 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:37:51.892 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:37:51.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:51.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:37:51.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:37:51.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-06-03T06:37:51.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:37:51.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-06-03T06:37:51.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:37:51.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:37:51.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T06:37:51.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:37:51.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:37:51.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:37:51.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:37:51.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:37:51.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:37:51.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:37:51.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:37:51.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:37:51.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:37:51.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:37:51.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:37:51.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:37:51.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:37:51.926 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:37:51.926 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:37:51.926 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:37:51.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:51.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:37:51.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:37:51.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:37:51.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:37:51.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:37:51.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:37:51.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:37:51.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:37:51.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:37:51.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:37:51.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:37:51.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:37:51.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:37:51.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:37:51.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0755bf50-f90e-4658-8e1d-0c2c1567b84d 2024-06-03T06:37:51.928 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 0755bf50-f90e-4658-8e1d-0c2c1567b84d 2024-06-03T06:37:51.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0755bf50-f90e-4658-8e1d-0c2c1567b84d' 2024-06-03T06:37:51.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:37:51.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/ZF1mgm5BOBAAHqghQ0xVZG5UbYWCy737xg== 2024-06-03T06:37:51.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/ZF1mgm5BOBAAHqghQ0xVZG5UbYWCy737xg=="}' 2024-06-03T06:37:51.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0755bf50-f90e-4658-8e1d-0c2c1567b84d -i td/osd-scrub-repair/0/new.json 2024-06-03T06:37:52.156 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:37:52.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T06:37:52.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/ZF1mgm5BOBAAHqghQ0xVZG5UbYWCy737xg== --osd-uuid 0755bf50-f90e-4658-8e1d-0c2c1567b84d 2024-06-03T06:37:52.198 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:37:52.197+0000 7fedee8ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:52.206 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:37:52.206+0000 7fedee8ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:37:52.207+0000 7fedee8ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:37:52.207+0000 7fedee8ac640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:37:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:37:52.207+0000 7fedee8ac640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:37:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:37:52.207+0000 7fedee8ac640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:37:52.290 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:37:52.289+0000 7fedee8ac640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T06:37:57.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T06:37:57.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:37:57.822 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:37:57.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T06:37:57.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:37:58.201 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:37:58.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T06:37:58.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:37:58.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:37:58.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:37:58.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:37:58.228 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:37:58.228+0000 7f7697ea2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:58.247 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:37:58.246+0000 7f7697ea2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:58.248 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:37:58.248+0000 7f7697ea2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:37:58.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T06:37:58.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:37:58.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:37:58.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:37:58.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:37:58.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:58.495 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:37:58.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:37:58.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:58.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:37:58.762 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:37:58.762+0000 7f7697ea2640 -1 Falling back to public interface 2024-06-03T06:37:58.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:37:59.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:37:59.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:37:59.787 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:37:59.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:37:59.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:37:59.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:00.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:01.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:01.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:01.087 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:38:01.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:38:01.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:01.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:01.114 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:01.112+0000 7f7697ea2640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:38:01.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:02.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:02.515 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:38:02.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:02.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:38:02.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:02.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:02.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:03.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:03.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:03.860 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:38:03.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:38:03.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:03.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:04.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:05.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:05.162 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:38:05.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:05.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:38:05.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:05.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:05.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:06.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:06.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:38:06.462 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:38:06.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:06.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:06.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:07.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:07.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:07.812 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:38:07.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:38:07.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:07.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:08.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:09.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:09.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:09.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:38:09.111 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:38:09.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:09.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:09.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:10.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:10.413 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:38:10.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:10.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:38:10.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:10.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:10.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:11.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:11.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:11.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:38:11.719 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:38:11.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:11.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:13.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:13.242 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:38:13.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:13.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:38:13.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:13.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:13.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:14.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:14.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:14.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:38:14.834 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:38:14.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:14.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:15.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:16.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:16.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:16.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:38:16.540 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:38:16.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:16.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:17.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:18.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:18.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:18.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:38:18.214 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:38:18.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:18.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:18.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:19.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:19.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:19.987 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:38:19.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:38:19.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:19.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:20.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:21.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:21.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:21.698 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:38:21.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:38:21.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:21.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:22.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:23.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:23.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:23.110 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:38:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:38:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:38:23.480 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/130367846,v1:127.0.0.1:6803/130367846] [v2:127.0.0.1:6804/130367846,v1:127.0.0.1:6805/130367846] exists,up 0755bf50-f90e-4658-8e1d-0c2c1567b84d 2024-06-03T06:38:23.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:38:23.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:38:23.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:38:23.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:38:23.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-06-03T06:38:23.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:38:23.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:38:23.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T06:38:23.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:38:23.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:38:23.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:38:23.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:38:23.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:38:23.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:38:23.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:38:23.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:38:23.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:38:23.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:38:23.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:38:23.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:38:23.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:38:23.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:38:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:38:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:38:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:38:23.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:38:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:38:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:38:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:38:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:38:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:38:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:38:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:38:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:38:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:38:23.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:38:23.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:38:23.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:38:23.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:38:23.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:38:23.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c656b644-c085-4489-bc0d-fc0e56b66605 2024-06-03T06:38:23.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c656b644-c085-4489-bc0d-fc0e56b66605' 2024-06-03T06:38:23.488 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 c656b644-c085-4489-bc0d-fc0e56b66605 2024-06-03T06:38:23.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:38:23.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfZF1m4Az4HRAAPFUvpSsJ5On2zKVje6uM0Q== 2024-06-03T06:38:23.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfZF1m4Az4HRAAPFUvpSsJ5On2zKVje6uM0Q=="}' 2024-06-03T06:38:23.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c656b644-c085-4489-bc0d-fc0e56b66605 -i td/osd-scrub-repair/1/new.json 2024-06-03T06:38:23.886 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:38:23.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T06:38:23.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfZF1m4Az4HRAAPFUvpSsJ5On2zKVje6uM0Q== --osd-uuid c656b644-c085-4489-bc0d-fc0e56b66605 2024-06-03T06:38:23.927 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:23.927+0000 7f2019db2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:38:23.971 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:23.970+0000 7f2019db2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:38:23.972 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:23.971+0000 7f2019db2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:38:23.972 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:23.972+0000 7f2019db2640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:38:23.972 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:23.972+0000 7f2019db2640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:38:23.972 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:23.972+0000 7f2019db2640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:38:24.045 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:24.045+0000 7f2019db2640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T06:38:29.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T06:38:29.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:38:29.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T06:38:29.821 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:38:29.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:38:30.198 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:38:30.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T06:38:30.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:38:30.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:38:30.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:38:30.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:38:30.226 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:30.226+0000 7faf30463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:38:30.244 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:30.243+0000 7faf30463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:38:30.245 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:30.245+0000 7faf30463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:38:30.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T06:38:30.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:38:30.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:38:30.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:38:30.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:38:30.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:30.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:38:30.522 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:38:30.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:30.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:30.777 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:30.777+0000 7faf30463640 -1 Falling back to public interface 2024-06-03T06:38:30.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:31.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:31.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:31.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:38:31.812 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:38:31.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:31.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:32.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:33.097 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:33.096+0000 7faf30463640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:38:33.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:33.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:33.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:38:33.112 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:38:33.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:33.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:33.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:34.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:34.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:34.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:38:34.418 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:38:34.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:34.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:34.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:35.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:35.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:35.766 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:38:35.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:38:35.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:35.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:36.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:37.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:37.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:37.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:38:37.172 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:38:37.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:37.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:37.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:38.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:38.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:38.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:38:38.526 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:38:38.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:38.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:38.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:39.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:39.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:39.831 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:38:39.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:38:39.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:39.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:40.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:41.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:41.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:41.134 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:38:41.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:38:41.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:41.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:41.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:42.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:42.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:42.484 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:38:42.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:38:42.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:42.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:42.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:43.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:43.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:43.788 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:38:43.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:38:43.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:43.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:44.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:45.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:45.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:45.131 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:38:45.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:38:45.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:45.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:45.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:46.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:46.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:46.534 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:38:46.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:38:46.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:46.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:47.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:48.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:48.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:48.395 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:38:48.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:38:48.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:48.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:49.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:50.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:50.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:50.065 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:38:50.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:38:50.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:50.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:51.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:52.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:52.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:52.062 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:38:52.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:38:52.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:52.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:52.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:53.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:53.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:53.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:38:53.747 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:38:53.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:53.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:54.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:38:55.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:38:55.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:38:55.151 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:38:55.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:38:55.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:38:55.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:38:55.485 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2367642224,v1:127.0.0.1:6811/2367642224] [v2:127.0.0.1:6812/2367642224,v1:127.0.0.1:6813/2367642224] exists,up c656b644-c085-4489-bc0d-fc0e56b66605 2024-06-03T06:38:55.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:38:55.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:38:55.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:38:55.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:38:55.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-06-03T06:38:55.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:38:55.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:38:55.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T06:38:55.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:38:55.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:38:55.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:38:55.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:38:55.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:38:55.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:38:55.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:38:55.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:38:55.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:38:55.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:38:55.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:38:55.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:38:55.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:38:55.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:38:55.490 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:38:55.490 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:38:55.491 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:38:55.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:38:55.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:38:55.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:38:55.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:38:55.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:38:55.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:38:55.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:38:55.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:38:55.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:38:55.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:38:55.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:38:55.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:38:55.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:38:55.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:38:55.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:38:55.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fbb5c31f-544b-4704-bc7e-3668f16e15ba 2024-06-03T06:38:55.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fbb5c31f-544b-4704-bc7e-3668f16e15ba' 2024-06-03T06:38:55.492 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 fbb5c31f-544b-4704-bc7e-3668f16e15ba 2024-06-03T06:38:55.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:38:55.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/ZF1m1DA7HhAAKDytxkln0kukJo1PE0GFFQ== 2024-06-03T06:38:55.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/ZF1m1DA7HhAAKDytxkln0kukJo1PE0GFFQ=="}' 2024-06-03T06:38:55.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fbb5c31f-544b-4704-bc7e-3668f16e15ba -i td/osd-scrub-repair/2/new.json 2024-06-03T06:38:55.851 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:38:55.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-03T06:38:55.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD/ZF1m1DA7HhAAKDytxkln0kukJo1PE0GFFQ== --osd-uuid fbb5c31f-544b-4704-bc7e-3668f16e15ba 2024-06-03T06:38:55.893 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:55.891+0000 7fea49c1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:38:55.927 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:55.926+0000 7fea49c1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:38:55.928 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:55.926+0000 7fea49c1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:38:55.929 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:55.927+0000 7fea49c1a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:38:55.929 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:55.927+0000 7fea49c1a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:38:55.929 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:55.927+0000 7fea49c1a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:38:55.994 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:38:55.993+0000 7fea49c1a640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T06:39:01.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-03T06:39:01.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:39:01.798 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T06:39:01.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T06:39:01.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:39:02.183 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:39:02.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T06:39:02.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:39:02.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:39:02.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:39:02.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:39:02.210 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:02.210+0000 7f8798714640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:39:02.218 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:02.218+0000 7f8798714640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:39:02.220 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:02.219+0000 7f8798714640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:39:02.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T06:39:02.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:39:02.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:39:02.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:39:02.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:39:02.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:02.494 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:39:02.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:39:02.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:02.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:02.751 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:02.751+0000 7f8798714640 -1 Falling back to public interface 2024-06-03T06:39:02.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:03.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:03.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:03.789 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:39:03.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:39:03.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:03.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:04.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:05.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:05.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:05.109 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:39:05.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:39:05.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:05.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:05.163 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:05.162+0000 7f8798714640 -1 osd.2 0 log_to_monitors true 2024-06-03T06:39:05.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:06.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:06.439 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:39:06.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:39:06.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:06.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:06.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:07.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:07.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:07.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:39:07.820 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:39:07.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:07.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:08.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:09.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:09.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:09.165 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:39:09.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:39:09.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:09.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:09.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:10.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:10.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:10.483 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:39:10.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:39:10.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:10.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:10.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:11.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:11.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:11.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:39:11.797 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:39:11.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:11.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:13.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:13.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:13.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:39:13.145 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:39:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:13.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:14.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:14.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:14.510 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:39:14.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:39:14.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:14.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:14.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:15.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:15.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:15.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:39:15.809 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:39:15.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:15.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:16.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:17.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:17.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:17.198 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:39:17.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:39:17.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:17.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:17.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:18.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:18.616 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:39:18.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:18.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:39:18.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:18.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:19.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:20.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:20.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:20.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:39:20.262 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:39:20.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:20.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:20.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:21.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:21.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:21.982 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:39:21.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:39:21.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:21.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:22.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:23.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:23.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:23.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:39:23.482 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:39:23.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:23.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:24.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:25.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:25.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:25.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:39:25.164 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:39:25.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:25.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:25.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:26.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:26.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:26.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:39:26.848 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:39:26.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:26.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:27.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:28.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:28.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:28.176 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:39:28.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:39:28.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:28.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:39:28.476 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1959892959,v1:127.0.0.1:6819/1959892959] [v2:127.0.0.1:6820/1959892959,v1:127.0.0.1:6821/1959892959] exists,up fbb5c31f-544b-4704-bc7e-3668f16e15ba 2024-06-03T06:39:28.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:39:28.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:39:28.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:39:28.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:39:28.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-06-03T06:39:28.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:39:28.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:39:28.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-03T06:39:28.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:39:28.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:39:28.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:39:28.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:39:28.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:39:28.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:39:28.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:39:28.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:39:28.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:39:28.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:39:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:39:28.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:39:28.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:39:28.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:39:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:39:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:39:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:39:28.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:39:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:39:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:39:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:39:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:39:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:39:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:39:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:39:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:39:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:39:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:39:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:39:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:39:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:39:28.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:39:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a251791b-8f01-466a-8f68-ddc09d9720c7 2024-06-03T06:39:28.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 a251791b-8f01-466a-8f68-ddc09d9720c7' 2024-06-03T06:39:28.484 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 a251791b-8f01-466a-8f68-ddc09d9720c7 2024-06-03T06:39:28.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:39:28.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAgZV1m2ZetHRAAQurW3od6OqMpRDppQW+Ecw== 2024-06-03T06:39:28.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAgZV1m2ZetHRAAQurW3od6OqMpRDppQW+Ecw=="}' 2024-06-03T06:39:28.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a251791b-8f01-466a-8f68-ddc09d9720c7 -i td/osd-scrub-repair/3/new.json 2024-06-03T06:39:28.877 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:39:28.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-06-03T06:39:28.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAgZV1m2ZetHRAAQurW3od6OqMpRDppQW+Ecw== --osd-uuid a251791b-8f01-466a-8f68-ddc09d9720c7 2024-06-03T06:39:28.919 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:28.918+0000 7fc6b5a43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:39:28.962 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:28.962+0000 7fc6b5a43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:39:28.963 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:28.963+0000 7fc6b5a43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:39:28.964 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:28.963+0000 7fc6b5a43640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-03T06:39:28.964 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:28.963+0000 7fc6b5a43640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-03T06:39:28.964 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:28.963+0000 7fc6b5a43640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-03T06:39:29.037 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:29.036+0000 7fc6b5a43640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-06-03T06:39:34.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-06-03T06:39:34.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:39:34.756 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-06-03T06:39:34.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-03T06:39:34.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:39:35.143 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-06-03T06:39:35.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-03T06:39:35.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:39:35.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:39:35.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:39:35.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:39:35.171 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:35.170+0000 7fa337b6e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:39:35.173 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:35.173+0000 7fa337b6e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:39:35.174 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:35.174+0000 7fa337b6e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:39:35.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-03T06:39:35.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:39:35.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:39:35.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:39:35.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:39:35.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:35.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:39:35.442 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:39:35.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:35.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:35.692 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:35.692+0000 7fa337b6e640 -1 Falling back to public interface 2024-06-03T06:39:35.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:36.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:36.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:36.730 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:39:36.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:39:36.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:36.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:37.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:37.959 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:39:37.958+0000 7fa337b6e640 -1 osd.3 0 log_to_monitors true 2024-06-03T06:39:38.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:38.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:38.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:39:38.029 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:39:38.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:38.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:38.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:39.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:39.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:39.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:39:39.402 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:39:39.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:39.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:39.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:40.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:40.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:40.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:39:40.743 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:39:40.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:40.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:41.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:42.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:42.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:42.065 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:39:42.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:39:42.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:42.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:42.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:43.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:43.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:43.368 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:39:43.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:39:43.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:43.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:43.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:44.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:44.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:44.734 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:39:44.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:39:44.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:44.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:45.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:46.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:46.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:46.070 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:39:46.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:39:46.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:46.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:46.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:47.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:47.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:47.371 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:39:47.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:39:47.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:47.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:47.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:48.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:48.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:48.711 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:39:48.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:39:48.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:48.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:49.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:50.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:50.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:50.128 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:39:50.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:39:50.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:50.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:51.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:52.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:52.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:52.006 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:39:52.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:39:52.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:52.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:52.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:53.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:53.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:53.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:39:53.408 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:39:53.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:53.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:53.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:54.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:54.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:54.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:39:54.893 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:39:54.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:54.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:55.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:56.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:56.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:56.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:39:56.610 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:39:56.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:56.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:57.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:39:58.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:39:58.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:39:58.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:39:58.263 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:39:58.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:39:58.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:39:59.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:00.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:00.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:00.097 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:40:00.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:40:00.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:00.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:40:00.400 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2821524709,v1:127.0.0.1:6827/2821524709] [v2:127.0.0.1:6828/2821524709,v1:127.0.0.1:6829/2821524709] exists,up a251791b-8f01-466a-8f68-ddc09d9720c7 2024-06-03T06:40:00.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:40:00.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:40:00.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:40:00.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:40:00.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-06-03T06:40:00.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:40:00.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:40:00.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-06-03T06:40:00.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:40:00.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-06-03T06:40:00.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:40:00.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:40:00.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:40:00.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:40:00.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-06-03T06:40:00.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-06-03T06:40:00.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:40:00.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:40:00.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:40:00.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:40:00.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:40:00.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:40:00.406 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:40:00.406 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:40:00.406 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:40:00.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:40:00.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:40:00.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:40:00.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:40:00.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:40:00.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:40:00.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:40:00.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:40:00.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:40:00.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:40:00.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:40:00.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:40:00.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:40:00.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-06-03T06:40:00.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:40:00.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=38a68488-9e92-4130-9041-38e84708466b 2024-06-03T06:40:00.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 38a68488-9e92-4130-9041-38e84708466b' 2024-06-03T06:40:00.408 INFO:tasks.workunit.client.0.smithi039.stdout:add osd4 38a68488-9e92-4130-9041-38e84708466b 2024-06-03T06:40:00.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:40:00.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBAZV1mwIwrGRAA+A/fDPSPdQufiWaK/EBLlA== 2024-06-03T06:40:00.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBAZV1mwIwrGRAA+A/fDPSPdQufiWaK/EBLlA=="}' 2024-06-03T06:40:00.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 38a68488-9e92-4130-9041-38e84708466b -i td/osd-scrub-repair/4/new.json 2024-06-03T06:40:00.797 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:40:00.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-06-03T06:40:00.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAZV1mwIwrGRAA+A/fDPSPdQufiWaK/EBLlA== --osd-uuid 38a68488-9e92-4130-9041-38e84708466b 2024-06-03T06:40:00.838 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:00.837+0000 7fd3bf98e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:40:00.873 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:00.872+0000 7fd3bf98e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:40:00.874 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:00.873+0000 7fd3bf98e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:40:00.874 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:00.874+0000 7fd3bf98e640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-03T06:40:00.874 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:00.874+0000 7fd3bf98e640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-03T06:40:00.874 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:00.874+0000 7fd3bf98e640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-03T06:40:00.948 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:00.947+0000 7fd3bf98e640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-06-03T06:40:06.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-06-03T06:40:06.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:40:06.687 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd4 key to auth repository 2024-06-03T06:40:06.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-06-03T06:40:06.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:40:07.102 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.4 2024-06-03T06:40:07.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-06-03T06:40:07.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:40:07.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:40:07.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:40:07.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:40:07.130 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:07.129+0000 7f7fd0d33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:40:07.132 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:07.132+0000 7f7fd0d33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:40:07.133 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:07.133+0000 7f7fd0d33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:40:07.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-06-03T06:40:07.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:40:07.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-06-03T06:40:07.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:40:07.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:40:07.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:07.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:40:07.393 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:40:07.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:07.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:07.648 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:07.648+0000 7f7fd0d33640 -1 Falling back to public interface 2024-06-03T06:40:07.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:08.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:08.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:08.681 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:40:08.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:40:08.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:08.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:08.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:09.982 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:40:09.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:40:09.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:09.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:10.000 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:09.999+0000 7f7fd0d33640 -1 osd.4 0 log_to_monitors true 2024-06-03T06:40:10.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:11.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:11.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:11.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:40:11.409 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:40:11.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:11.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:11.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:12.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:12.620 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:40:12.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:12.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:40:12.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:12.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:13.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:14.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:14.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:14.064 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:40:14.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:40:14.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:14.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:14.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:15.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:15.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:15.369 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:40:15.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:40:15.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:15.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:15.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:16.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:16.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:16.670 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:40:16.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:40:16.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:16.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:16.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:17.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:17.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:17.969 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:40:17.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:40:17.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:17.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:18.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:19.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:19.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:19.271 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:40:19.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:40:19.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:19.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:19.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:20.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:20.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:20.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:40:20.603 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:40:20.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:20.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:21.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:22.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:22.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:22.002 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:40:22.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:40:22.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:22.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:22.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:23.647 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:40:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:40:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:24.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:25.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:25.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:25.176 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:40:25.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:40:25.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:25.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:25.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:26.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:26.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:26.542 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:40:26.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:40:26.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:26.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:27.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:28.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:28.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:28.204 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:40:28.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:40:28.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:28.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:28.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:29.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:29.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:29.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:40:29.982 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:40:29.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:29.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:30.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:31.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:31.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:31.451 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:40:31.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:40:31.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:31.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:31.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:32.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:32.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:32.850 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:40:32.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:40:32.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:32.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:40:33.153 INFO:tasks.workunit.client.0.smithi039.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2602415529,v1:127.0.0.1:6835/2602415529] [v2:127.0.0.1:6836/2602415529,v1:127.0.0.1:6837/2602415529] exists,up 38a68488-9e92-4130-9041-38e84708466b 2024-06-03T06:40:33.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:40:33.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:40:33.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:40:33.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:40:33.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-06-03T06:40:33.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:40:33.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:40:33.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-06-03T06:40:33.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:40:33.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:40:33.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:40:33.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:40:33.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:40:33.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:40:33.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-03T06:40:33.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-03T06:40:33.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:40:33.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:40:33.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:40:33.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:40:33.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:40:33.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:40:33.159 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:40:33.159 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:40:33.159 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:40:33.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:40:33.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:40:33.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:40:33.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:40:33.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:40:33.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:40:33.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:40:33.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:40:33.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:40:33.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:40:33.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:40:33.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:40:33.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:40:33.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-06-03T06:40:33.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:40:33.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c849e020-3670-4066-b986-5db1f468a8d7 2024-06-03T06:40:33.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 c849e020-3670-4066-b986-5db1f468a8d7' 2024-06-03T06:40:33.161 INFO:tasks.workunit.client.0.smithi039.stdout:add osd5 c849e020-3670-4066-b986-5db1f468a8d7 2024-06-03T06:40:33.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:40:33.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhZV1mIlFvChAAFyM1SIw8xfcuhmMf88GIeg== 2024-06-03T06:40:33.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhZV1mIlFvChAAFyM1SIw8xfcuhmMf88GIeg=="}' 2024-06-03T06:40:33.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c849e020-3670-4066-b986-5db1f468a8d7 -i td/osd-scrub-repair/5/new.json 2024-06-03T06:40:33.524 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:40:33.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-06-03T06:40:33.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhZV1mIlFvChAAFyM1SIw8xfcuhmMf88GIeg== --osd-uuid c849e020-3670-4066-b986-5db1f468a8d7 2024-06-03T06:40:33.565 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:33.565+0000 7f73daa0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:40:33.592 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:33.591+0000 7f73daa0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:40:33.593 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:33.592+0000 7f73daa0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:40:33.593 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:33.593+0000 7f73daa0e640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-03T06:40:33.593 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:33.593+0000 7f73daa0e640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-03T06:40:33.593 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:33.593+0000 7f73daa0e640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-03T06:40:33.650 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:33.650+0000 7f73daa0e640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-06-03T06:40:39.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-06-03T06:40:39.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:40:39.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-06-03T06:40:39.074 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd5 key to auth repository 2024-06-03T06:40:39.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:40:39.449 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.5 2024-06-03T06:40:39.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-06-03T06:40:39.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:40:39.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:40:39.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:40:39.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:40:39.476 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:39.475+0000 7fb3fee25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:40:39.478 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:39.477+0000 7fb3fee25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:40:39.479 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:39.478+0000 7fb3fee25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:40:39.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-06-03T06:40:39.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:40:39.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-03T06:40:39.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:40:39.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:40:39.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:39.739 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:40:39.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:40:39.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:39.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:40:40.004 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:40.003+0000 7fb3fee25640 -1 Falling back to public interface 2024-06-03T06:40:40.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:41.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:41.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:41.027 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:40:41.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:40:41.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:41.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:40:41.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:42.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:42.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:42.327 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:40:42.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:40:42.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:42.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:40:42.404 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:40:42.402+0000 7fb3fee25640 -1 osd.5 0 log_to_monitors true 2024-06-03T06:40:42.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:43.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:43.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:43.647 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:40:43.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:40:43.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:43.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:40:43.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:44.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:44.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:44.949 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:40:44.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:40:44.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:44.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:40:45.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:46.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:46.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:46.286 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:40:46.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:40:46.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:46.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:40:46.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:47.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:47.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:47.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:40:47.588 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:40:47.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:47.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:40:47.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:48.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:48.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:48.889 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:40:48.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:40:48.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:48.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:40:49.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:50.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:50.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:50.260 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:40:50.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:40:50.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:50.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:40:50.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:51.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:51.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:51.563 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:40:51.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:40:51.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:51.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:40:51.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:52.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:52.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:52.867 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:40:52.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:40:52.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:52.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:40:53.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:54.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:54.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:54.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:40:54.203 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:40:54.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:54.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:40:54.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:55.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:55.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:55.694 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:40:55.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:40:55.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:55.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:40:56.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:57.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:57.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:57.396 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:40:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:40:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:40:58.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:40:59.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:40:59.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:40:59.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:40:59.024 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:40:59.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:40:59.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:40:59.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:00.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:00.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:00.544 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:41:00.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:41:00.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:00.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:41:01.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:02.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:02.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:02.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:41:02.348 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:41:02.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:02.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:41:02.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:03.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:03.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:03.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:41:03.976 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:41:03.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:03.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:41:04.279 INFO:tasks.workunit.client.0.smithi039.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2318094050,v1:127.0.0.1:6843/2318094050] [v2:127.0.0.1:6844/2318094050,v1:127.0.0.1:6845/2318094050] exists,up c849e020-3670-4066-b986-5db1f468a8d7 2024-06-03T06:41:04.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:41:04.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:41:04.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:41:04.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:41:04.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-06-03T06:41:04.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:41:04.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:41:04.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-06-03T06:41:04.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:41:04.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-06-03T06:41:04.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:41:04.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:41:04.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:41:04.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:41:04.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-06-03T06:41:04.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-06-03T06:41:04.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:41:04.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:41:04.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:41:04.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:41:04.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:41:04.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:41:04.284 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:41:04.284 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:41:04.284 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:41:04.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:41:04.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:41:04.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:41:04.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:41:04.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:41:04.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:41:04.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:41:04.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:41:04.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:41:04.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:41:04.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:41:04.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:41:04.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:41:04.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-06-03T06:41:04.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:41:04.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3375dc5f-839d-4873-86db-20b54a54953a 2024-06-03T06:41:04.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 3375dc5f-839d-4873-86db-20b54a54953a' 2024-06-03T06:41:04.286 INFO:tasks.workunit.client.0.smithi039.stdout:add osd6 3375dc5f-839d-4873-86db-20b54a54953a 2024-06-03T06:41:04.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:41:04.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCAZV1mkzH1ERAAC6FW7IeRq/yQzPt12CxX6Q== 2024-06-03T06:41:04.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCAZV1mkzH1ERAAC6FW7IeRq/yQzPt12CxX6Q=="}' 2024-06-03T06:41:04.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3375dc5f-839d-4873-86db-20b54a54953a -i td/osd-scrub-repair/6/new.json 2024-06-03T06:41:04.661 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:41:04.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-06-03T06:41:04.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCAZV1mkzH1ERAAC6FW7IeRq/yQzPt12CxX6Q== --osd-uuid 3375dc5f-839d-4873-86db-20b54a54953a 2024-06-03T06:41:04.702 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:04.702+0000 7f3c243a7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:41:04.729 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:04.728+0000 7f3c243a7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:41:04.730 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:04.729+0000 7f3c243a7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:41:04.731 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:04.730+0000 7f3c243a7640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-03T06:41:04.731 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:04.730+0000 7f3c243a7640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-03T06:41:04.731 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:04.730+0000 7f3c243a7640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-03T06:41:04.787 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:04.787+0000 7f3c243a7640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-06-03T06:41:10.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-06-03T06:41:10.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:41:10.718 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd6 key to auth repository 2024-06-03T06:41:10.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-06-03T06:41:10.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:41:11.111 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.6 2024-06-03T06:41:11.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-06-03T06:41:11.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:41:11.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:41:11.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:41:11.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:41:11.139 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:11.138+0000 7f1fbc36e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:41:11.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:11.140+0000 7f1fbc36e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:41:11.142 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:11.141+0000 7f1fbc36e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:41:11.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-06-03T06:41:11.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:41:11.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-06-03T06:41:11.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:41:11.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:41:11.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:11.403 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:41:11.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:41:11.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:11.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:11.657 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:11.656+0000 7f1fbc36e640 -1 Falling back to public interface 2024-06-03T06:41:11.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:12.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:12.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:12.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:41:12.698 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:41:12.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:12.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:12.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:13.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:13.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:13.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:41:13.996 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:41:13.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:13.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:14.010 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:14.009+0000 7f1fbc36e640 -1 osd.6 0 log_to_monitors true 2024-06-03T06:41:14.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:15.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:15.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:15.344 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:41:15.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:41:15.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:15.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:15.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:16.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:16.649 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:41:16.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:16.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:41:16.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:16.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:17.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:18.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:18.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:18.060 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:41:18.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:41:18.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:18.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:18.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:19.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:19.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:19.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:41:19.361 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:41:19.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:19.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:19.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:20.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:20.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:20.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:41:20.792 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:41:20.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:20.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:21.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:22.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:22.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:22.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:41:22.094 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:41:22.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:22.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:22.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:23.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:23.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:23.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:41:23.395 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:41:23.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:23.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:23.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:24.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:24.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:24.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:41:24.698 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:41:24.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:24.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:25.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:25.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:25.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:41:25.999 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:41:25.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:26.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:26.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:27.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:27.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:27.468 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:41:27.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:41:27.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:27.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:27.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:28.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:28.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:28.927 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:41:28.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:41:28.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:28.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:29.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:30.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:30.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:30.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:41:30.361 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:41:30.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:30.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:31.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:32.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:32.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:32.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:41:32.307 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:41:32.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:32.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:33.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:34.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:34.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:34.051 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:41:34.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:41:34.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:34.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:34.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:35.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:35.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:35.545 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:41:35.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:41:35.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:35.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:36.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:37.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:37.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:37.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:41:37.424 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:41:37.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:37.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:41:37.724 INFO:tasks.workunit.client.0.smithi039.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/804483462,v1:127.0.0.1:6851/804483462] [v2:127.0.0.1:6852/804483462,v1:127.0.0.1:6853/804483462] exists,up 3375dc5f-839d-4873-86db-20b54a54953a 2024-06-03T06:41:37.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:41:37.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:41:37.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:41:37.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:41:37.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-06-03T06:41:37.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:41:37.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:41:37.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-06-03T06:41:37.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:41:37.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-06-03T06:41:37.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:41:37.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:41:37.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:41:37.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:41:37.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-06-03T06:41:37.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-06-03T06:41:37.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:41:37.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:41:37.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:41:37.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:41:37.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:41:37.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:41:37.729 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:41:37.729 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:41:37.729 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:41:37.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:41:37.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:41:37.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:41:37.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:41:37.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:41:37.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:41:37.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:41:37.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:41:37.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:41:37.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:41:37.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:41:37.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:41:37.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:41:37.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-06-03T06:41:37.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:41:37.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a4cb28bf-273d-46f5-b59e-b0db06cae780 2024-06-03T06:41:37.731 INFO:tasks.workunit.client.0.smithi039.stdout:add osd7 a4cb28bf-273d-46f5-b59e-b0db06cae780 2024-06-03T06:41:37.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 a4cb28bf-273d-46f5-b59e-b0db06cae780' 2024-06-03T06:41:37.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:41:37.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQChZV1mW/97LBAACwM6fuOoqGPKnLSZP+kzhw== 2024-06-03T06:41:37.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQChZV1mW/97LBAACwM6fuOoqGPKnLSZP+kzhw=="}' 2024-06-03T06:41:37.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a4cb28bf-273d-46f5-b59e-b0db06cae780 -i td/osd-scrub-repair/7/new.json 2024-06-03T06:41:38.103 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:41:38.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-06-03T06:41:38.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQChZV1mW/97LBAACwM6fuOoqGPKnLSZP+kzhw== --osd-uuid a4cb28bf-273d-46f5-b59e-b0db06cae780 2024-06-03T06:41:38.144 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:38.144+0000 7f71e2415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:41:38.180 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:38.179+0000 7f71e2415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:41:38.181 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:38.180+0000 7f71e2415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:41:38.181 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:38.181+0000 7f71e2415640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-03T06:41:38.181 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:38.181+0000 7f71e2415640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-03T06:41:38.181 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:38.181+0000 7f71e2415640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-03T06:41:38.254 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:38.254+0000 7f71e2415640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-06-03T06:41:44.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-06-03T06:41:44.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:41:44.798 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd7 key to auth repository 2024-06-03T06:41:44.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-06-03T06:41:44.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:41:45.174 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.7 2024-06-03T06:41:45.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-06-03T06:41:45.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:41:45.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:41:45.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:41:45.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:41:45.202 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:45.202+0000 7f0f72010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:41:45.204 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:45.204+0000 7f0f72010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:41:45.205 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:45.205+0000 7f0f72010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:41:45.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-06-03T06:41:45.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:41:45.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-06-03T06:41:45.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:41:45.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:41:45.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:45.471 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:41:45.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:41:45.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:45.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:41:45.721 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:45.721+0000 7f0f72010640 -1 Falling back to public interface 2024-06-03T06:41:45.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:46.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:46.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:46.765 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:41:46.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:41:46.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:46.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:41:47.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:48.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:48.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:48.062 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:41:48.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:41:48.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:48.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:41:48.091 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:41:48.090+0000 7f0f72010640 -1 osd.7 0 log_to_monitors true 2024-06-03T06:41:48.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:49.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:49.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:49.368 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:41:49.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:41:49.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:49.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:41:49.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:50.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:50.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:50.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:41:50.775 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:41:50.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:50.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:41:51.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:52.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:52.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:52.130 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:41:52.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:41:52.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:52.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:41:52.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:53.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:53.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:53.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:41:53.435 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:41:53.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:53.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:41:53.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:54.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:54.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:54.749 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:41:54.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:41:54.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:54.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:41:55.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:56.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:56.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:56.142 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:41:56.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:41:56.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:56.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:41:56.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:57.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:57.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:57.442 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:41:57.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:41:57.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:57.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:41:57.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:41:58.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:41:58.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:41:58.757 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:41:58.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:41:58.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:41:58.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:41:59.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:00.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:00.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:00.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:42:00.122 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:42:00.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:00.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:42:00.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:01.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:01.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:01.423 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:42:01.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:42:01.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:01.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:42:02.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:03.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:03.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:03.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:42:03.053 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:42:03.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:03.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:42:03.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:04.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:04.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:04.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:42:04.638 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:42:04.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:04.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:42:05.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:06.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:06.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:06.432 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:42:06.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:42:06.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:06.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:42:07.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:42:08.419 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:42:08.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:08.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:42:09.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:10.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:10.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:10.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:42:10.080 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:42:10.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:10.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:42:10.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:11.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:11.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:11.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:42:11.478 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:42:11.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:11.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:42:11.877 INFO:tasks.workunit.client.0.smithi039.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1262434041,v1:127.0.0.1:6859/1262434041] [v2:127.0.0.1:6860/1262434041,v1:127.0.0.1:6861/1262434041] exists,up a4cb28bf-273d-46f5-b59e-b0db06cae780 2024-06-03T06:42:11.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:42:11.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:42:11.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:42:11.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:42:11.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-06-03T06:42:11.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:42:11.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:42:11.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-06-03T06:42:11.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:42:11.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-06-03T06:42:11.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:42:11.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:42:11.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:42:11.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:42:11.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-06-03T06:42:11.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-06-03T06:42:11.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:42:11.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:42:11.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:42:11.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:42:11.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:42:11.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:42:11.883 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:42:11.883 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:42:11.884 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:42:11.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:42:11.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:42:11.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:42:11.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:42:11.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:42:11.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:42:11.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:42:11.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:42:11.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:42:11.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:42:11.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:42:11.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:42:11.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:42:11.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-06-03T06:42:11.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:42:11.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8f1ec34e-dd51-4957-83ed-9a1026dac763 2024-06-03T06:42:11.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 8f1ec34e-dd51-4957-83ed-9a1026dac763' 2024-06-03T06:42:11.885 INFO:tasks.workunit.client.0.smithi039.stdout:add osd8 8f1ec34e-dd51-4957-83ed-9a1026dac763 2024-06-03T06:42:11.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:42:11.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDZV1mE3SqNRAAVzLLphHYHFrlJIwSRGAZlQ== 2024-06-03T06:42:11.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDZV1mE3SqNRAAVzLLphHYHFrlJIwSRGAZlQ=="}' 2024-06-03T06:42:11.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8f1ec34e-dd51-4957-83ed-9a1026dac763 -i td/osd-scrub-repair/8/new.json 2024-06-03T06:42:12.243 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:42:12.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-06-03T06:42:12.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDZV1mE3SqNRAAVzLLphHYHFrlJIwSRGAZlQ== --osd-uuid 8f1ec34e-dd51-4957-83ed-9a1026dac763 2024-06-03T06:42:12.284 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:12.283+0000 7fac94b2d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:42:12.346 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:12.346+0000 7fac94b2d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:42:12.347 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:12.347+0000 7fac94b2d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:42:12.348 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:12.347+0000 7fac94b2d640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-03T06:42:12.348 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:12.347+0000 7fac94b2d640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-03T06:42:12.348 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:12.347+0000 7fac94b2d640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-03T06:42:12.402 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:12.402+0000 7fac94b2d640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-06-03T06:42:18.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-06-03T06:42:18.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:42:18.882 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd8 key to auth repository 2024-06-03T06:42:18.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-06-03T06:42:18.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:42:19.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-06-03T06:42:19.260 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.8 2024-06-03T06:42:19.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:42:19.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:42:19.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:42:19.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:42:19.288 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:19.286+0000 7fd06f811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:42:19.290 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:19.288+0000 7fd06f811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:42:19.291 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:19.289+0000 7fd06f811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:42:19.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-06-03T06:42:19.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:42:19.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-06-03T06:42:19.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:42:19.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:42:19.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:19.553 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:42:19.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:42:19.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:19.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:19.805 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:19.804+0000 7fd06f811640 -1 Falling back to public interface 2024-06-03T06:42:19.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:20.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:20.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:20.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:42:20.842 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:42:20.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:20.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:21.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:22.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:22.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:22.143 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:42:22.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:42:22.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:22.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:22.335 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:22.333+0000 7fd06f811640 -1 osd.8 0 log_to_monitors true 2024-06-03T06:42:22.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:23.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:23.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:23.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:42:23.447 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:42:23.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:23.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:23.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:24.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:24.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:24.748 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:42:24.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:42:24.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:24.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:25.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:26.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:26.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:26.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:42:26.158 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:42:26.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:26.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:26.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:27.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:27.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:27.510 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:42:27.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:42:27.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:27.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:27.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:28.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:28.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:28.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:42:28.813 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:42:28.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:28.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:29.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:30.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:30.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:30.168 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:42:30.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:42:30.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:30.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:30.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:31.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:31.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:31.470 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:42:31.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:42:31.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:31.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:31.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:32.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:32.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:32.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:42:32.772 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:42:32.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:32.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:33.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:34.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:34.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:34.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:42:34.144 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:42:34.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:34.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:34.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:35.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:35.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:35.449 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:42:35.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:42:35.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:35.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:35.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:36.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:36.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:36.985 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:42:36.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:42:36.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:36.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:37.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:38.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:38.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:38.421 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:42:38.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:42:38.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:38.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:39.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:40.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:40.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:40.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:42:40.159 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:42:40.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:40.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:40.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:41.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:41.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:41.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:42:41.486 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:42:41.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:41.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:42.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:43.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:43.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:43.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:42:43.222 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:42:43.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:43.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:43.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:44.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:44.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:44.900 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:42:44.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:42:44.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:44.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:45.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:46.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:46.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:46.240 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:42:46.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:42:46.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:46.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:42:46.538 INFO:tasks.workunit.client.0.smithi039.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/3676036245,v1:127.0.0.1:6867/3676036245] [v2:127.0.0.1:6868/3676036245,v1:127.0.0.1:6869/3676036245] exists,up 8f1ec34e-dd51-4957-83ed-9a1026dac763 2024-06-03T06:42:46.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:42:46.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:42:46.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:42:46.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:42:46.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-06-03T06:42:46.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:42:46.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:42:46.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-06-03T06:42:46.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:42:46.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:42:46.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:42:46.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:42:46.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:42:46.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:42:46.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-03T06:42:46.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-03T06:42:46.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:42:46.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:42:46.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:42:46.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:42:46.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:42:46.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:42:46.543 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:42:46.543 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:42:46.543 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:42:46.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:42:46.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:42:46.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:42:46.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:42:46.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:42:46.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:42:46.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:42:46.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:42:46.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:42:46.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:42:46.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:42:46.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:42:46.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:42:46.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-06-03T06:42:46.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:42:46.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2a84bc6a-1558-4d3e-8c00-d333d070e6fc 2024-06-03T06:42:46.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 2a84bc6a-1558-4d3e-8c00-d333d070e6fc' 2024-06-03T06:42:46.545 INFO:tasks.workunit.client.0.smithi039.stdout:add osd9 2a84bc6a-1558-4d3e-8c00-d333d070e6fc 2024-06-03T06:42:46.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:42:46.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDmZV1mEdVlIRAAC185kksmVXSx2Pl1LpuFjQ== 2024-06-03T06:42:46.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDmZV1mEdVlIRAAC185kksmVXSx2Pl1LpuFjQ=="}' 2024-06-03T06:42:46.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2a84bc6a-1558-4d3e-8c00-d333d070e6fc -i td/osd-scrub-repair/9/new.json 2024-06-03T06:42:46.932 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:42:46.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-06-03T06:42:46.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDmZV1mEdVlIRAAC185kksmVXSx2Pl1LpuFjQ== --osd-uuid 2a84bc6a-1558-4d3e-8c00-d333d070e6fc 2024-06-03T06:42:46.974 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:46.973+0000 7f99457f6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:42:47.034 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:47.033+0000 7f99457f6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:42:47.035 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:47.034+0000 7f99457f6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:42:47.035 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:47.035+0000 7f99457f6640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-03T06:42:47.035 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:47.035+0000 7f99457f6640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-03T06:42:47.035 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:47.035+0000 7f99457f6640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-03T06:42:47.117 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:47.116+0000 7f99457f6640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-06-03T06:42:53.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-06-03T06:42:53.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:42:53.437 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd9 key to auth repository 2024-06-03T06:42:53.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-06-03T06:42:53.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:42:53.815 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.9 2024-06-03T06:42:53.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-06-03T06:42:53.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:42:53.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:42:53.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:42:53.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:42:53.843 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:53.842+0000 7f4b70422640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:42:53.845 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:53.844+0000 7f4b70422640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:42:53.846 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:53.845+0000 7f4b70422640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:42:54.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-06-03T06:42:54.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:42:54.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-03T06:42:54.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:42:54.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:42:54.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:54.107 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:42:54.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:42:54.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:54.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:42:54.364 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:54.364+0000 7f4b70422640 -1 Falling back to public interface 2024-06-03T06:42:54.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:55.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:55.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:55.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:42:55.399 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:42:55.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:55.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:42:55.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:56.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:56.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:56.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:42:56.702 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:42:56.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:56.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:42:56.707 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:42:56.707+0000 7f4b70422640 -1 osd.9 0 log_to_monitors true 2024-06-03T06:42:57.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:58.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:58.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:58.025 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:42:58.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:42:58.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:58.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:42:58.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:42:59.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:42:59.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:42:59.398 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:42:59.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:42:59.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:42:59.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:42:59.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:00.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:00.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:00.720 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:43:00.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:43:00.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:00.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:43:01.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:02.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:02.021 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:43:02.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:02.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:43:02.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:02.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:43:02.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:03.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:03.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:03.408 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:43:03.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:43:03.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:03.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:43:03.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:04.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:04.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:04.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:43:04.796 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:43:04.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:04.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:43:05.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:06.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:06.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:06.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:43:06.129 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:43:06.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:06.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:43:06.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:07.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:07.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:07.430 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:43:07.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:43:07.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:07.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:43:07.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:08.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:08.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:08.863 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:43:08.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:43:08.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:08.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:43:09.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:10.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:10.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:10.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:43:10.164 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:43:10.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:10.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:43:10.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:11.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:11.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:11.609 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:43:11.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:43:11.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:11.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:43:12.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:13.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:13.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:13.011 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:43:13.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:43:13.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:13.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:43:13.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:14.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:14.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:14.740 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:43:14.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:43:14.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:14.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:43:15.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:16.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:16.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:16.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:43:16.526 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:43:16.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:16.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:43:17.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:18.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:18.296 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:43:18.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:18.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:43:18.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:18.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:43:18.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:19.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:19.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:19.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:43:19.732 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:43:19.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:19.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:43:20.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:21.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:21.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:21.299 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:43:21.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:43:21.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:21.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:43:21.608 INFO:tasks.workunit.client.0.smithi039.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/3093910756,v1:127.0.0.1:6875/3093910756] [v2:127.0.0.1:6876/3093910756,v1:127.0.0.1:6877/3093910756] exists,up 2a84bc6a-1558-4d3e-8c00-d333d070e6fc 2024-06-03T06:43:21.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:43:21.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:43:21.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:43:21.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-06-03T06:43:21.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T06:43:21.891 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-06-03T06:43:21.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T06:43:21.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T06:43:22.501 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T06:43:22.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:43:23.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T06:43:26.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-06-03T06:43:26.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:43:26.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:43:26.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:43:26.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:43:26.772 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:43:26.772 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:43:26.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:43:26.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:43:26.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:43:26.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:43:26.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:43:26.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:43:26.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:43:26.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:43:26.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:43:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:43:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:43:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:43:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:43:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-06-03T06:43:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:5 2024-06-03T06:43:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:6 2024-06-03T06:43:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:7 2024-06-03T06:43:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:8 2024-06-03T06:43:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:9' 2024-06-03T06:43:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:43:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:27.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:43:27.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836544 2024-06-03T06:43:27.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836544 2024-06-03T06:43:27.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544' 2024-06-03T06:43:27.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:27.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:43:27.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673020 2024-06-03T06:43:27.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673020 2024-06-03T06:43:27.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020' 2024-06-03T06:43:27.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:27.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:43:27.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509494 2024-06-03T06:43:27.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509494 2024-06-03T06:43:27.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509494' 2024-06-03T06:43:27.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:27.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:43:27.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345968 2024-06-03T06:43:27.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345968 2024-06-03T06:43:27.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509494 3-85899345968' 2024-06-03T06:43:27.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:27.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:43:27.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182440 2024-06-03T06:43:27.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182440 2024-06-03T06:43:27.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509494 3-85899345968 4-107374182440' 2024-06-03T06:43:27.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:27.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:43:28.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018915 2024-06-03T06:43:28.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018915 2024-06-03T06:43:28.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509494 3-85899345968 4-107374182440 5-128849018915' 2024-06-03T06:43:28.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:28.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:43:28.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855387 2024-06-03T06:43:28.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855387 2024-06-03T06:43:28.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509494 3-85899345968 4-107374182440 5-128849018915 6-150323855387' 2024-06-03T06:43:28.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:28.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:43:28.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691862 2024-06-03T06:43:28.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691862 2024-06-03T06:43:28.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509494 3-85899345968 4-107374182440 5-128849018915 6-150323855387 7-171798691862' 2024-06-03T06:43:28.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:28.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:43:28.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528333 2024-06-03T06:43:28.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528333 2024-06-03T06:43:28.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509494 3-85899345968 4-107374182440 5-128849018915 6-150323855387 7-171798691862 8-193273528333' 2024-06-03T06:43:28.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:28.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:43:28.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364812 2024-06-03T06:43:28.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364812 2024-06-03T06:43:28.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509494 3-85899345968 4-107374182440 5-128849018915 6-150323855387 7-171798691862 8-193273528333 9-214748364812' 2024-06-03T06:43:28.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:28.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836544 2024-06-03T06:43:28.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:28.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:43:28.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836544 2024-06-03T06:43:28.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:28.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836544 2024-06-03T06:43:28.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-06-03T06:43:28.445 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836544 2024-06-03T06:43:28.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:43:28.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836544 2024-06-03T06:43:28.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:28.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673020 2024-06-03T06:43:28.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:28.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:43:28.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673020 2024-06-03T06:43:28.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:28.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673020 2024-06-03T06:43:28.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2024-06-03T06:43:28.736 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673020 2024-06-03T06:43:28.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:43:29.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673020 -lt 42949673020 2024-06-03T06:43:29.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:29.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509494 2024-06-03T06:43:29.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:29.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:43:29.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509494 2024-06-03T06:43:29.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:29.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509494 2024-06-03T06:43:29.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509494' 2024-06-03T06:43:29.030 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509494 2024-06-03T06:43:29.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:43:29.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509494 -lt 64424509494 2024-06-03T06:43:29.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:29.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345968 2024-06-03T06:43:29.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:29.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:43:29.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345968 2024-06-03T06:43:29.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:29.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345968 2024-06-03T06:43:29.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345968' 2024-06-03T06:43:29.331 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345968 2024-06-03T06:43:29.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:43:29.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345968 -lt 85899345968 2024-06-03T06:43:29.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:29.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182440 2024-06-03T06:43:29.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:29.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:43:29.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182440 2024-06-03T06:43:29.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:29.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182440 2024-06-03T06:43:29.666 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182440 2024-06-03T06:43:29.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182440' 2024-06-03T06:43:29.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:43:29.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182440 -lt 107374182440 2024-06-03T06:43:29.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:29.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018915 2024-06-03T06:43:29.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:29.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:43:29.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018915 2024-06-03T06:43:29.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:29.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018915 2024-06-03T06:43:29.958 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018915 2024-06-03T06:43:29.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018915' 2024-06-03T06:43:29.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:43:30.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018916 -lt 128849018915 2024-06-03T06:43:30.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:30.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855387 2024-06-03T06:43:30.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:30.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:43:30.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855387 2024-06-03T06:43:30.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:30.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855387 2024-06-03T06:43:30.253 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.6 seq 150323855387 2024-06-03T06:43:30.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855387' 2024-06-03T06:43:30.254 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:43:30.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855387 -lt 150323855387 2024-06-03T06:43:30.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:30.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691862 2024-06-03T06:43:30.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:30.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:43:30.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691862 2024-06-03T06:43:30.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:30.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691862 2024-06-03T06:43:30.566 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.7 seq 171798691862 2024-06-03T06:43:30.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691862' 2024-06-03T06:43:30.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:43:30.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691862 -lt 171798691862 2024-06-03T06:43:30.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:30.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528333 2024-06-03T06:43:30.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:30.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:43:30.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528333 2024-06-03T06:43:30.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:30.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528333 2024-06-03T06:43:30.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528333' 2024-06-03T06:43:30.860 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.8 seq 193273528333 2024-06-03T06:43:30.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:43:31.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528333 -lt 193273528333 2024-06-03T06:43:31.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:31.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364812 2024-06-03T06:43:31.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:31.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:43:31.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364812 2024-06-03T06:43:31.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:31.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364812 2024-06-03T06:43:31.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364812' 2024-06-03T06:43:31.163 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.9 seq 214748364812 2024-06-03T06:43:31.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:43:31.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364812 -lt 214748364812 2024-06-03T06:43:31.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:43:31.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:43:31.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:43:31.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T06:43:31.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:43:31.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:43:31.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:43:31.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:43:31.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:43:31.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:43:31.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:43:32.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:43:32.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:43:32.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:43:32.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:43:32.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T06:43:32.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:43:32.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:43:32.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-06-03T06:43:32.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-03T06:43:32.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-03T06:43:32.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-06-03T06:43:32.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-03T06:43:32.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-06-03T06:43:32.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-03T06:43:32.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-03T06:43:34.049 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'ecpool' created 2024-06-03T06:43:34.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:43:35.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-06-03T06:43:35.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-03T06:43:35.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:43:35.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:43:35.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:43:35.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:43:35.066 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:43:35.066 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:43:35.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:43:35.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:43:35.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:43:35.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:43:35.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:43:35.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:43:35.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:43:35.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:43:35.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:43:35.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:43:35.468 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:43:35.468 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:43:35.468 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:43:35.468 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-06-03T06:43:35.468 INFO:tasks.workunit.client.0.smithi039.stderr:5 2024-06-03T06:43:35.469 INFO:tasks.workunit.client.0.smithi039.stderr:6 2024-06-03T06:43:35.469 INFO:tasks.workunit.client.0.smithi039.stderr:7 2024-06-03T06:43:35.469 INFO:tasks.workunit.client.0.smithi039.stderr:8 2024-06-03T06:43:35.469 INFO:tasks.workunit.client.0.smithi039.stderr:9' 2024-06-03T06:43:35.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:43:35.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:35.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:43:35.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836547 2024-06-03T06:43:35.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836547 2024-06-03T06:43:35.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547' 2024-06-03T06:43:35.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:35.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:43:35.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673023 2024-06-03T06:43:35.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673023 2024-06-03T06:43:35.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673023' 2024-06-03T06:43:35.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:35.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:43:35.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509496 2024-06-03T06:43:35.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509496 2024-06-03T06:43:35.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509496' 2024-06-03T06:43:35.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:35.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:43:35.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345971 2024-06-03T06:43:35.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345971 2024-06-03T06:43:35.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509496 3-85899345971' 2024-06-03T06:43:35.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:35.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:43:36.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182442 2024-06-03T06:43:36.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182442 2024-06-03T06:43:36.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509496 3-85899345971 4-107374182442' 2024-06-03T06:43:36.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:36.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:43:36.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018918 2024-06-03T06:43:36.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018918 2024-06-03T06:43:36.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509496 3-85899345971 4-107374182442 5-128849018918' 2024-06-03T06:43:36.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:36.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:43:36.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855389 2024-06-03T06:43:36.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855389 2024-06-03T06:43:36.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509496 3-85899345971 4-107374182442 5-128849018918 6-150323855389' 2024-06-03T06:43:36.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:36.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:43:36.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691864 2024-06-03T06:43:36.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691864 2024-06-03T06:43:36.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509496 3-85899345971 4-107374182442 5-128849018918 6-150323855389 7-171798691864' 2024-06-03T06:43:36.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:36.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:43:36.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528336 2024-06-03T06:43:36.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528336 2024-06-03T06:43:36.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509496 3-85899345971 4-107374182442 5-128849018918 6-150323855389 7-171798691864 8-193273528336' 2024-06-03T06:43:36.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:36.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:43:36.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364815 2024-06-03T06:43:36.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364815 2024-06-03T06:43:36.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509496 3-85899345971 4-107374182442 5-128849018918 6-150323855389 7-171798691864 8-193273528336 9-214748364815' 2024-06-03T06:43:36.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:36.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836547 2024-06-03T06:43:36.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:36.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:43:36.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836547 2024-06-03T06:43:36.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:36.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836547 2024-06-03T06:43:36.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-06-03T06:43:36.674 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836547 2024-06-03T06:43:36.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:43:36.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836547 -lt 21474836547 2024-06-03T06:43:36.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:36.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673023 2024-06-03T06:43:36.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:36.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:43:36.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673023 2024-06-03T06:43:36.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:36.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673023 2024-06-03T06:43:36.965 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673023 2024-06-03T06:43:36.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2024-06-03T06:43:36.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:43:37.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673023 -lt 42949673023 2024-06-03T06:43:37.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:37.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509496 2024-06-03T06:43:37.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:37.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:43:37.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509496 2024-06-03T06:43:37.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:37.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509496 2024-06-03T06:43:37.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2024-06-03T06:43:37.261 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509496 2024-06-03T06:43:37.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:43:37.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509496 -lt 64424509496 2024-06-03T06:43:37.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:37.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345971 2024-06-03T06:43:37.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:37.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:43:37.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345971 2024-06-03T06:43:37.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:37.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345971 2024-06-03T06:43:37.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345971' 2024-06-03T06:43:37.556 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345971 2024-06-03T06:43:37.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:43:37.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345971 -lt 85899345971 2024-06-03T06:43:37.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:37.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182442 2024-06-03T06:43:37.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:37.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:43:37.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182442 2024-06-03T06:43:37.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:37.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182442 2024-06-03T06:43:37.848 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182442 2024-06-03T06:43:37.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182442' 2024-06-03T06:43:37.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:43:38.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182441 -lt 107374182442 2024-06-03T06:43:38.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:43:39.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:43:39.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:43:39.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182443 -lt 107374182442 2024-06-03T06:43:39.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:39.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018918 2024-06-03T06:43:39.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:39.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:43:39.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018918 2024-06-03T06:43:39.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:39.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018918 2024-06-03T06:43:39.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018918' 2024-06-03T06:43:39.457 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018918 2024-06-03T06:43:39.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:43:39.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018918 -lt 128849018918 2024-06-03T06:43:39.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:39.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855389 2024-06-03T06:43:39.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:39.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:43:39.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855389 2024-06-03T06:43:39.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:39.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855389 2024-06-03T06:43:39.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855389' 2024-06-03T06:43:39.749 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.6 seq 150323855389 2024-06-03T06:43:39.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:43:40.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855390 -lt 150323855389 2024-06-03T06:43:40.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:40.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691864 2024-06-03T06:43:40.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:40.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:43:40.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691864 2024-06-03T06:43:40.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:40.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691864 2024-06-03T06:43:40.058 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.7 seq 171798691864 2024-06-03T06:43:40.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691864' 2024-06-03T06:43:40.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:43:40.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691865 -lt 171798691864 2024-06-03T06:43:40.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:40.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528336 2024-06-03T06:43:40.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:40.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:43:40.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528336 2024-06-03T06:43:40.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:40.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528336 2024-06-03T06:43:40.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528336' 2024-06-03T06:43:40.350 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.8 seq 193273528336 2024-06-03T06:43:40.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:43:40.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528336 -lt 193273528336 2024-06-03T06:43:40.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:40.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364815 2024-06-03T06:43:40.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:40.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:43:40.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364815 2024-06-03T06:43:40.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:40.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364815 2024-06-03T06:43:40.642 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.9 seq 214748364815 2024-06-03T06:43:40.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364815' 2024-06-03T06:43:40.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:43:40.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364815 -lt 214748364815 2024-06-03T06:43:40.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:43:40.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:43:40.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:43:41.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:43:41.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:43:41.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:43:41.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:43:41.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:43:41.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:43:41.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:43:41.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:43:41.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:43:41.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:43:41.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:43:41.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:43:41.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-03T06:43:41.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-03T06:43:41.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:43:41.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-03T06:43:41.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:43:41.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:43:42.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:43:42.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:43:42.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:43:42.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:43:42.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:43:42.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:43:42.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:43:42.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:43:42.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:43:42.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:43:42.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:43:42.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:43:42.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:43:42.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:43:42.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:43:42.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-03T06:43:42.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-06-03T06:43:42.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-03T06:43:42.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-06-03T06:43:42.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-06-03T06:43:42.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T06:43:42.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T06:43:42.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-03T06:43:42.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T06:43:42.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T06:43:42.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T06:43:43.166 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T06:43:43.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T06:43:43.778 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T06:43:43.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T06:43:43.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T06:43:43.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-03T06:43:44.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-06-03T06:43:44.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-03T06:43:44.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:43:44.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:43:44.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:43:44.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-06-03T06:43:44.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-06-03T06:43:44.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-03T06:43:44.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-06-03T06:43:44.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-06-03T06:43:44.633 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:43:44.633 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T06:43:44.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-06-03T06:43:44.968 INFO:tasks.workunit.client.0.smithi039.stderr:5 2024-06-03T06:43:44.968 INFO:tasks.workunit.client.0.smithi039.stderr:9 2024-06-03T06:43:44.968 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:43:44.968 INFO:tasks.workunit.client.0.smithi039.stderr:6 2024-06-03T06:43:44.968 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:43:44.969 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:43:44.969 INFO:tasks.workunit.client.0.smithi039.stderr:7' 2024-06-03T06:43:44.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-06-03T06:43:44.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-06-03T06:43:44.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-06-03T06:43:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-06-03T06:43:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-06-03T06:43:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-06-03T06:43:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-03T06:43:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-03T06:43:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-06-03T06:43:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:43:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:43:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:43:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:43:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:43:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:43:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:43:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:43:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:43:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:43:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:43:44.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:43:44.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:43:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:43:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:43:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:43:45.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:43:45.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:43:45.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:43:45.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:43:45.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:43:45.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:43:45.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:43:45.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-03T06:43:47.244 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-06-03T06:43:47.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:43:47.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:43:47.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:43:47.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:43:47.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:43:47.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:43:47.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:43:47.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:43:47.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:43:47.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:43:47.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:43:47.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:43:47.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:43:47.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:43:47.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:43:47.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:43:47.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:43:47.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:43:47.763 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:43:47.763 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:43:47.763 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:43:47.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:43:47.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:43:47.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:43:47.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:43:47.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:43:47.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:43:47.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:43:47.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:43:47.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:43:47.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:43:47.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:43:47.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:43:47.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:43:47.765 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.3 2024-06-03T06:43:47.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:43:47.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:43:47.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:43:47.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:43:47.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:43:47.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:43:47.794 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:43:47.793+0000 7f1c8de2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:43:47.796 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:43:47.796+0000 7f1c8de2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:43:47.798 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:43:47.797+0000 7f1c8de2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:43:48.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:43:48.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:43:48.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:43:48.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:43:48.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:43:48.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:48.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:43:48.069 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:43:48.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:48.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:43:48.318 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:43:48.318+0000 7f1c8de2e640 -1 Falling back to public interface 2024-06-03T06:43:48.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:49.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:49.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:49.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:43:49.367 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:43:49.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:49.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:43:49.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:50.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:50.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:50.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:43:50.671 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:43:50.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:50.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:43:50.758 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:43:50.758+0000 7f1c8de2e640 -1 osd.3 68 log_to_monitors true 2024-06-03T06:43:50.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:43:51.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:43:51.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:43:51.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:43:51.976 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:43:51.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:43:51.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:43:52.345 INFO:tasks.workunit.client.0.smithi039.stderr:osd.3 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [20,68) [v2:127.0.0.1:6826/2823541809,v1:127.0.0.1:6827/2823541809] [v2:127.0.0.1:6828/2823541809,v1:127.0.0.1:6829/2823541809] exists,up a251791b-8f01-466a-8f68-ddc09d9720c7 2024-06-03T06:43:52.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:43:52.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:43:52.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:43:52.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:43:52.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:43:52.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:43:52.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:43:52.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:43:52.347 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:43:52.347 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:43:52.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:43:52.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:43:52.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:43:52.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:43:52.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:43:52.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:43:52.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:43:52.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:43:52.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:43:52.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:43:52.732 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:43:52.732 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:43:52.732 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:43:52.732 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-06-03T06:43:52.733 INFO:tasks.workunit.client.0.smithi039.stderr:5 2024-06-03T06:43:52.733 INFO:tasks.workunit.client.0.smithi039.stderr:6 2024-06-03T06:43:52.733 INFO:tasks.workunit.client.0.smithi039.stderr:7 2024-06-03T06:43:52.733 INFO:tasks.workunit.client.0.smithi039.stderr:8 2024-06-03T06:43:52.733 INFO:tasks.workunit.client.0.smithi039.stderr:9' 2024-06-03T06:43:52.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:43:52.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:52.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:43:52.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836551 2024-06-03T06:43:52.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836551 2024-06-03T06:43:52.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551' 2024-06-03T06:43:52.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:52.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:43:52.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673027 2024-06-03T06:43:52.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673027 2024-06-03T06:43:52.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673027' 2024-06-03T06:43:52.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:52.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:43:53.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509501 2024-06-03T06:43:53.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509501 2024-06-03T06:43:53.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673027 2-64424509501' 2024-06-03T06:43:53.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:53.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:43:53.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645314 2024-06-03T06:43:53.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645314 2024-06-03T06:43:53.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673027 2-64424509501 3-309237645314' 2024-06-03T06:43:53.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:53.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:43:53.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182447 2024-06-03T06:43:53.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182447 2024-06-03T06:43:53.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673027 2-64424509501 3-309237645314 4-107374182447' 2024-06-03T06:43:53.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:53.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:43:53.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018922 2024-06-03T06:43:53.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018922 2024-06-03T06:43:53.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673027 2-64424509501 3-309237645314 4-107374182447 5-128849018922' 2024-06-03T06:43:53.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:53.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:43:53.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855394 2024-06-03T06:43:53.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855394 2024-06-03T06:43:53.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673027 2-64424509501 3-309237645314 4-107374182447 5-128849018922 6-150323855394' 2024-06-03T06:43:53.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:53.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:43:53.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691869 2024-06-03T06:43:53.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691869 2024-06-03T06:43:53.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673027 2-64424509501 3-309237645314 4-107374182447 5-128849018922 6-150323855394 7-171798691869' 2024-06-03T06:43:53.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:53.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:43:53.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528340 2024-06-03T06:43:53.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528340 2024-06-03T06:43:53.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673027 2-64424509501 3-309237645314 4-107374182447 5-128849018922 6-150323855394 7-171798691869 8-193273528340' 2024-06-03T06:43:53.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:43:53.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:43:53.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364819 2024-06-03T06:43:53.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364819 2024-06-03T06:43:53.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673027 2-64424509501 3-309237645314 4-107374182447 5-128849018922 6-150323855394 7-171798691869 8-193273528340 9-214748364819' 2024-06-03T06:43:53.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:53.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836551 2024-06-03T06:43:53.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:53.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:43:53.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836551 2024-06-03T06:43:53.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:53.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836551 2024-06-03T06:43:53.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836551' 2024-06-03T06:43:53.847 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836551 2024-06-03T06:43:53.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:43:54.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836551 2024-06-03T06:43:54.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:43:55.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:43:55.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:43:55.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836552 -lt 21474836551 2024-06-03T06:43:55.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:55.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673027 2024-06-03T06:43:55.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:55.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:43:55.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673027 2024-06-03T06:43:55.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:55.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673027 2024-06-03T06:43:55.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673027' 2024-06-03T06:43:55.460 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949673027 2024-06-03T06:43:55.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:43:55.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673027 -lt 42949673027 2024-06-03T06:43:55.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:55.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509501 2024-06-03T06:43:55.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:55.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:43:55.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509501 2024-06-03T06:43:55.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:55.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509501 2024-06-03T06:43:55.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509501' 2024-06-03T06:43:55.758 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509501 2024-06-03T06:43:55.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:43:56.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509501 -lt 64424509501 2024-06-03T06:43:56.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:56.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645314 2024-06-03T06:43:56.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:56.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:43:56.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645314 2024-06-03T06:43:56.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:56.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645314 2024-06-03T06:43:56.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645314' 2024-06-03T06:43:56.051 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 309237645314 2024-06-03T06:43:56.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:43:56.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645315 -lt 309237645314 2024-06-03T06:43:56.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:56.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182447 2024-06-03T06:43:56.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:56.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:43:56.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182447 2024-06-03T06:43:56.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:56.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182447 2024-06-03T06:43:56.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182447' 2024-06-03T06:43:56.343 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.4 seq 107374182447 2024-06-03T06:43:56.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:43:56.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182447 -lt 107374182447 2024-06-03T06:43:56.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:56.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018922 2024-06-03T06:43:56.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:56.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:43:56.657 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018922 2024-06-03T06:43:56.657 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:56.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018922 2024-06-03T06:43:56.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018922' 2024-06-03T06:43:56.659 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.5 seq 128849018922 2024-06-03T06:43:56.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:43:56.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018923 -lt 128849018922 2024-06-03T06:43:56.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:56.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855394 2024-06-03T06:43:56.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:56.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:43:56.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855394 2024-06-03T06:43:56.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:56.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855394 2024-06-03T06:43:56.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855394' 2024-06-03T06:43:56.951 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.6 seq 150323855394 2024-06-03T06:43:56.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:43:57.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855394 -lt 150323855394 2024-06-03T06:43:57.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:57.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691869 2024-06-03T06:43:57.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:57.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:43:57.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691869 2024-06-03T06:43:57.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:57.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691869 2024-06-03T06:43:57.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691869' 2024-06-03T06:43:57.279 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.7 seq 171798691869 2024-06-03T06:43:57.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:43:57.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691869 -lt 171798691869 2024-06-03T06:43:57.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:57.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528340 2024-06-03T06:43:57.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:57.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:43:57.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528340 2024-06-03T06:43:57.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:57.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528340 2024-06-03T06:43:57.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528340' 2024-06-03T06:43:57.570 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.8 seq 193273528340 2024-06-03T06:43:57.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:43:57.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528341 -lt 193273528340 2024-06-03T06:43:57.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:43:57.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364819 2024-06-03T06:43:57.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:43:57.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:43:57.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364819 2024-06-03T06:43:57.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:43:57.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364819 2024-06-03T06:43:57.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364819' 2024-06-03T06:43:57.862 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.9 seq 214748364819 2024-06-03T06:43:57.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:43:58.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364819 -lt 214748364819 2024-06-03T06:43:58.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:43:58.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:43:58.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:43:58.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:43:58.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:43:58.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:43:58.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:43:58.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:43:58.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:43:58.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:43:58.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:43:58.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:43:58.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:43:58.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:43:58.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:43:59.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:43:59.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:43:59.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:43:59.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-03T06:43:59.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:43:59.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:43:59.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:43:59.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:43:59.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-03T06:43:59.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-03T06:43:59.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:43:59.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:43:59.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:43:59.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:43:59.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:43:59.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:43:59.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:43:33.780982+0000 2024-06-03T06:43:59.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:44:00.048 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:44:00.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:43:33.780982+0000 2024-06-03T06:44:00.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:44:00.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:43:33.780982+0000 2024-06-03T06:44:00.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:44:00.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:44:00.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:44:00.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:44:00.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:44:00.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:44:00.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:44:00.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:44:00.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:43:33.780982+0000 '>' 2024-06-03T06:43:33.780982+0000 2024-06-03T06:44:00.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:44:01.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:44:01.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:44:01.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:44:01.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:44:01.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:44:01.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:44:01.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:44:01.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:43:33.780982+0000 '>' 2024-06-03T06:43:33.780982+0000 2024-06-03T06:44:01.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:44:02.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:44:02.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:44:02.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:44:02.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:44:02.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:44:02.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:44:02.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:44:02.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:44:00.897568+0000 '>' 2024-06-03T06:43:33.780982+0000 2024-06-03T06:44:02.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:44:02.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:44:02.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:44:02.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:44:02.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-03T06:44:02.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:44:02.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:44:02.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:44:02.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:44:02.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-03T06:44:02.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:44:02.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-03T06:44:02.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:44:02.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:44:02.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:44:02.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:44:02.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:44:03.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:44:03.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:44:03.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:44:03.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:44:03.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-03T06:44:03.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:44:03.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:44:03.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-03T06:44:05.360 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:44:05.360 INFO:tasks.workunit.client.0.smithi039.stdout:hinfo_key 2024-06-03T06:44:05.360 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:44:05.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-03T06:44:05.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:44:05.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:44:05.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-03T06:44:05.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:44:05.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:44:05.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:44:05.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:44:05.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:44:05.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:44:05.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-03T06:44:05.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-03T06:44:05.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:44:05.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:44:05.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:44:05.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:44:05.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:44:05.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:44:05.629 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:44:05.629 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:44:05.629 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:44:05.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:44:05.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:44:05.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:44:05.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:44:05.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:44:05.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:44:05.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:44:05.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:44:05.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:44:05.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:44:05.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:44:05.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-03T06:44:05.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-03T06:44:05.631 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.9 2024-06-03T06:44:05.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:44:05.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-03T06:44:05.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-03T06:44:05.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:44:05.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:44:05.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:44:05.659 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:44:05.658+0000 7f366918e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:44:05.662 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:44:05.660+0000 7f366918e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:44:05.663 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:44:05.661+0000 7f366918e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:44:05.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-03T06:44:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:44:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-03T06:44:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:44:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:44:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:44:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:44:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:44:06.182 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:44:06.181+0000 7f366918e640 -1 Falling back to public interface 2024-06-03T06:44:06.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:44:07.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:44:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:44:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:44:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:07.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:44:07.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:44:08.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:44:08.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:08.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:44:08.525 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:44:08.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:08.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:44:08.662 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:44:08.661+0000 7f366918e640 -1 osd.9 73 log_to_monitors true 2024-06-03T06:44:08.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:44:09.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:44:09.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:09.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:44:09.875 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:44:09.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:09.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:44:10.212 INFO:tasks.workunit.client.0.smithi039.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 74 last_clean_interval [50,73) [v2:127.0.0.1:6874/2966797753,v1:127.0.0.1:6875/2966797753] [v2:127.0.0.1:6876/2966797753,v1:127.0.0.1:6877/2966797753] exists,up 2a84bc6a-1558-4d3e-8c00-d333d070e6fc 2024-06-03T06:44:10.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:44:10.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:44:10.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:44:10.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:44:10.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:44:10.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:44:10.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:44:10.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:44:10.213 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:44:10.213 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:44:10.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:44:10.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:44:10.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:44:10.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:44:10.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:44:10.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:44:10.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:44:10.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:44:10.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:44:10.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:44:10.654 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:44:10.655 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:44:10.655 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:44:10.655 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-06-03T06:44:10.655 INFO:tasks.workunit.client.0.smithi039.stderr:5 2024-06-03T06:44:10.655 INFO:tasks.workunit.client.0.smithi039.stderr:6 2024-06-03T06:44:10.655 INFO:tasks.workunit.client.0.smithi039.stderr:7 2024-06-03T06:44:10.655 INFO:tasks.workunit.client.0.smithi039.stderr:8 2024-06-03T06:44:10.655 INFO:tasks.workunit.client.0.smithi039.stderr:9' 2024-06-03T06:44:10.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:44:10.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:10.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:44:10.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836556 2024-06-03T06:44:10.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836556 2024-06-03T06:44:10.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556' 2024-06-03T06:44:10.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:10.821 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:44:10.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673032 2024-06-03T06:44:10.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673032 2024-06-03T06:44:10.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673032' 2024-06-03T06:44:10.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:10.929 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:44:11.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509505 2024-06-03T06:44:11.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509505 2024-06-03T06:44:11.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673032 2-64424509505' 2024-06-03T06:44:11.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:11.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:44:11.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645319 2024-06-03T06:44:11.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645319 2024-06-03T06:44:11.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673032 2-64424509505 3-309237645319' 2024-06-03T06:44:11.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:11.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:44:11.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182451 2024-06-03T06:44:11.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182451 2024-06-03T06:44:11.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673032 2-64424509505 3-309237645319 4-107374182451' 2024-06-03T06:44:11.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:11.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:44:11.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018927 2024-06-03T06:44:11.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018927 2024-06-03T06:44:11.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673032 2-64424509505 3-309237645319 4-107374182451 5-128849018927' 2024-06-03T06:44:11.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:11.383 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:44:11.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855398 2024-06-03T06:44:11.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855398 2024-06-03T06:44:11.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673032 2-64424509505 3-309237645319 4-107374182451 5-128849018927 6-150323855398' 2024-06-03T06:44:11.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:11.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:44:11.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691873 2024-06-03T06:44:11.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691873 2024-06-03T06:44:11.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673032 2-64424509505 3-309237645319 4-107374182451 5-128849018927 6-150323855398 7-171798691873' 2024-06-03T06:44:11.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:11.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:44:11.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528345 2024-06-03T06:44:11.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528345 2024-06-03T06:44:11.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673032 2-64424509505 3-309237645319 4-107374182451 5-128849018927 6-150323855398 7-171798691873 8-193273528345' 2024-06-03T06:44:11.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:11.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:44:11.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-06-03T06:44:11.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-06-03T06:44:11.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673032 2-64424509505 3-309237645319 4-107374182451 5-128849018927 6-150323855398 7-171798691873 8-193273528345 9-330712481794' 2024-06-03T06:44:11.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:11.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836556 2024-06-03T06:44:11.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:11.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:44:11.819 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836556 2024-06-03T06:44:11.819 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:11.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836556 2024-06-03T06:44:11.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-06-03T06:44:11.820 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836556 2024-06-03T06:44:11.820 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:44:12.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836555 -lt 21474836556 2024-06-03T06:44:12.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:44:13.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:44:13.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:44:13.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836556 -lt 21474836556 2024-06-03T06:44:13.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:13.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673032 2024-06-03T06:44:13.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:13.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:44:13.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673032 2024-06-03T06:44:13.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:13.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673032 2024-06-03T06:44:13.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673032' 2024-06-03T06:44:13.416 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949673032 2024-06-03T06:44:13.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:44:13.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673032 -lt 42949673032 2024-06-03T06:44:13.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:13.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509505 2024-06-03T06:44:13.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:13.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:44:13.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509505 2024-06-03T06:44:13.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:13.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509505 2024-06-03T06:44:13.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2024-06-03T06:44:13.752 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509505 2024-06-03T06:44:13.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:44:14.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509505 -lt 64424509505 2024-06-03T06:44:14.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:14.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645319 2024-06-03T06:44:14.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:14.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:44:14.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645319 2024-06-03T06:44:14.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:14.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645319 2024-06-03T06:44:14.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645319' 2024-06-03T06:44:14.044 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 309237645319 2024-06-03T06:44:14.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:44:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645319 -lt 309237645319 2024-06-03T06:44:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:14.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182451 2024-06-03T06:44:14.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:14.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:44:14.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182451 2024-06-03T06:44:14.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:14.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182451 2024-06-03T06:44:14.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2024-06-03T06:44:14.340 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.4 seq 107374182451 2024-06-03T06:44:14.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:44:14.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182452 -lt 107374182451 2024-06-03T06:44:14.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:14.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018927 2024-06-03T06:44:14.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:14.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:44:14.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018927 2024-06-03T06:44:14.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:14.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018927 2024-06-03T06:44:14.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018927' 2024-06-03T06:44:14.632 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.5 seq 128849018927 2024-06-03T06:44:14.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:44:14.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018927 -lt 128849018927 2024-06-03T06:44:14.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:14.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855398 2024-06-03T06:44:14.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:14.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:44:14.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855398 2024-06-03T06:44:14.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:14.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855398 2024-06-03T06:44:14.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855398' 2024-06-03T06:44:14.924 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.6 seq 150323855398 2024-06-03T06:44:14.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:44:15.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855399 -lt 150323855398 2024-06-03T06:44:15.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:15.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691873 2024-06-03T06:44:15.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:15.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:44:15.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691873 2024-06-03T06:44:15.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:15.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691873 2024-06-03T06:44:15.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691873' 2024-06-03T06:44:15.223 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.7 seq 171798691873 2024-06-03T06:44:15.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:44:15.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691874 -lt 171798691873 2024-06-03T06:44:15.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:15.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528345 2024-06-03T06:44:15.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:15.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:44:15.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528345 2024-06-03T06:44:15.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:15.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528345 2024-06-03T06:44:15.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528345' 2024-06-03T06:44:15.516 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.8 seq 193273528345 2024-06-03T06:44:15.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:44:15.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528345 -lt 193273528345 2024-06-03T06:44:15.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:15.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481794 2024-06-03T06:44:15.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:15.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:44:15.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481794 2024-06-03T06:44:15.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:15.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-06-03T06:44:15.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-06-03T06:44:15.812 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.9 seq 330712481794 2024-06-03T06:44:15.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:44:16.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-06-03T06:44:16.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:44:16.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:44:16.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:44:16.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:44:16.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:44:16.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:44:16.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:44:16.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:44:16.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:44:16.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:44:16.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:44:16.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:44:16.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:44:16.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:44:16.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:44:17.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:44:17.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:44:17.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:44:17.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:44:17.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:44:17.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-06-03T06:44:17.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-03T06:44:17.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-03T06:44:17.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-06-03T06:44:17.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:44:17.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:44:17.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:44:17.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-03T06:44:17.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:44:17.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:44:17.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:44:17.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:44:17.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-03T06:44:17.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:44:17.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-03T06:44:17.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:44:17.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:44:17.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:44:17.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:44:17.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:44:17.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:44:17.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:44:17.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:44:17.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:44:17.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-03T06:44:17.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:44:17.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:44:17.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-03T06:44:19.525 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-03T06:44:20.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-03T06:44:20.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:44:20.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:44:20.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-03T06:44:20.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:44:20.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:44:20.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:44:20.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:44:20.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:44:20.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:44:20.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-03T06:44:20.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-03T06:44:20.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:44:20.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:44:20.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:44:20.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:44:20.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:44:20.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:44:20.044 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:44:20.044 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:44:20.044 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:44:20.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:44:20.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:44:20.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:44:20.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:44:20.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:44:20.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:44:20.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:44:20.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:44:20.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:44:20.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:44:20.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:44:20.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-03T06:44:20.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-03T06:44:20.046 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.5 2024-06-03T06:44:20.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:44:20.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-03T06:44:20.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-03T06:44:20.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:44:20.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:44:20.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:44:20.075 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:44:20.075+0000 7fb33b8b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:44:20.078 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:44:20.078+0000 7fb33b8b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:44:20.080 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:44:20.079+0000 7fb33b8b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:44:20.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-03T06:44:20.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:44:20.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-03T06:44:20.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:44:20.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:44:20.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:20.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:44:20.364 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:44:20.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:20.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:44:20.594 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:44:20.594+0000 7fb33b8b1640 -1 Falling back to public interface 2024-06-03T06:44:20.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:44:21.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:44:21.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:21.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:44:21.655 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:44:21.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:21.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:44:21.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:44:22.952 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:44:22.952+0000 7fb33b8b1640 -1 osd.5 78 log_to_monitors true 2024-06-03T06:44:22.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:44:22.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:22.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:44:22.958 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:44:22.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:22.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:44:23.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:44:24.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:44:24.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:24.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:44:24.267 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:44:24.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:24.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:44:24.649 INFO:tasks.workunit.client.0.smithi039.stderr:osd.5 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [30,78) [v2:127.0.0.1:6842/1715238115,v1:127.0.0.1:6843/1715238115] [v2:127.0.0.1:6844/1715238115,v1:127.0.0.1:6845/1715238115] exists,up c849e020-3670-4066-b986-5db1f468a8d7 2024-06-03T06:44:24.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:44:24.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:44:24.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:44:24.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:44:24.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:44:24.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:44:24.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:44:24.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:44:24.650 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:44:24.651 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:44:24.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:44:24.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:44:24.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:44:24.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:44:24.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:44:24.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:44:24.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:44:24.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:44:24.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:44:25.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:44:25.247 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:44:25.247 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:44:25.247 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:44:25.247 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-06-03T06:44:25.247 INFO:tasks.workunit.client.0.smithi039.stderr:5 2024-06-03T06:44:25.247 INFO:tasks.workunit.client.0.smithi039.stderr:6 2024-06-03T06:44:25.247 INFO:tasks.workunit.client.0.smithi039.stderr:7 2024-06-03T06:44:25.247 INFO:tasks.workunit.client.0.smithi039.stderr:8 2024-06-03T06:44:25.248 INFO:tasks.workunit.client.0.smithi039.stderr:9' 2024-06-03T06:44:25.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:44:25.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:25.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:44:25.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836560 2024-06-03T06:44:25.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836560 2024-06-03T06:44:25.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560' 2024-06-03T06:44:25.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:25.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:44:25.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673035 2024-06-03T06:44:25.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673035 2024-06-03T06:44:25.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673035' 2024-06-03T06:44:25.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:25.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:44:25.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509509 2024-06-03T06:44:25.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509509 2024-06-03T06:44:25.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673035 2-64424509509' 2024-06-03T06:44:25.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:25.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:44:25.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645322 2024-06-03T06:44:25.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645322 2024-06-03T06:44:25.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673035 2-64424509509 3-309237645322' 2024-06-03T06:44:25.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:25.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:44:25.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182455 2024-06-03T06:44:25.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182455 2024-06-03T06:44:25.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673035 2-64424509509 3-309237645322 4-107374182455' 2024-06-03T06:44:25.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:25.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:44:26.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-06-03T06:44:26.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-06-03T06:44:26.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673035 2-64424509509 3-309237645322 4-107374182455 5-352187318274' 2024-06-03T06:44:26.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:26.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:44:26.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855402 2024-06-03T06:44:26.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855402 2024-06-03T06:44:26.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673035 2-64424509509 3-309237645322 4-107374182455 5-352187318274 6-150323855402' 2024-06-03T06:44:26.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:26.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:44:26.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691877 2024-06-03T06:44:26.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691877 2024-06-03T06:44:26.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673035 2-64424509509 3-309237645322 4-107374182455 5-352187318274 6-150323855402 7-171798691877' 2024-06-03T06:44:26.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:26.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:44:26.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528349 2024-06-03T06:44:26.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528349 2024-06-03T06:44:26.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673035 2-64424509509 3-309237645322 4-107374182455 5-352187318274 6-150323855402 7-171798691877 8-193273528349' 2024-06-03T06:44:26.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:26.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:44:26.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481798 2024-06-03T06:44:26.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481798 2024-06-03T06:44:26.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673035 2-64424509509 3-309237645322 4-107374182455 5-352187318274 6-150323855402 7-171798691877 8-193273528349 9-330712481798' 2024-06-03T06:44:26.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:26.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836560 2024-06-03T06:44:26.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:26.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:44:26.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836560 2024-06-03T06:44:26.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:26.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836560 2024-06-03T06:44:26.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836560' 2024-06-03T06:44:26.551 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836560 2024-06-03T06:44:26.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:44:26.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836560 -lt 21474836560 2024-06-03T06:44:26.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:26.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673035 2024-06-03T06:44:26.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:26.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:44:26.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673035 2024-06-03T06:44:26.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:26.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673035 2024-06-03T06:44:26.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-06-03T06:44:26.843 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949673035 2024-06-03T06:44:26.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:44:27.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673036 -lt 42949673035 2024-06-03T06:44:27.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:27.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509509 2024-06-03T06:44:27.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:27.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:44:27.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509509 2024-06-03T06:44:27.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:27.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509509 2024-06-03T06:44:27.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509509' 2024-06-03T06:44:27.170 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509509 2024-06-03T06:44:27.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:44:27.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509509 -lt 64424509509 2024-06-03T06:44:27.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:27.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645322 2024-06-03T06:44:27.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:27.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:44:27.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645322 2024-06-03T06:44:27.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:27.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645322 2024-06-03T06:44:27.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645322' 2024-06-03T06:44:27.505 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 309237645322 2024-06-03T06:44:27.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:44:27.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645323 -lt 309237645322 2024-06-03T06:44:27.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:27.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182455 2024-06-03T06:44:27.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:27.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:44:27.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182455 2024-06-03T06:44:27.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:27.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182455 2024-06-03T06:44:27.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182455' 2024-06-03T06:44:27.803 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.4 seq 107374182455 2024-06-03T06:44:27.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:44:28.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182455 -lt 107374182455 2024-06-03T06:44:28.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:28.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-352187318274 2024-06-03T06:44:28.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:28.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:44:28.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-352187318274 2024-06-03T06:44:28.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:28.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-06-03T06:44:28.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 352187318274' 2024-06-03T06:44:28.094 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.5 seq 352187318274 2024-06-03T06:44:28.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:44:28.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-06-03T06:44:28.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:28.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855402 2024-06-03T06:44:28.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:28.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:44:28.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855402 2024-06-03T06:44:28.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:28.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855402 2024-06-03T06:44:28.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855402' 2024-06-03T06:44:28.412 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.6 seq 150323855402 2024-06-03T06:44:28.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:44:28.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855403 -lt 150323855402 2024-06-03T06:44:28.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:28.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691877 2024-06-03T06:44:28.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:28.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:44:28.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691877 2024-06-03T06:44:28.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:28.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691877 2024-06-03T06:44:28.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691877' 2024-06-03T06:44:28.709 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.7 seq 171798691877 2024-06-03T06:44:28.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:44:29.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691878 -lt 171798691877 2024-06-03T06:44:29.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:29.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528349 2024-06-03T06:44:29.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:29.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:44:29.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528349 2024-06-03T06:44:29.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:29.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528349 2024-06-03T06:44:29.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528349' 2024-06-03T06:44:29.004 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.8 seq 193273528349 2024-06-03T06:44:29.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:44:29.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528349 -lt 193273528349 2024-06-03T06:44:29.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:29.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481798 2024-06-03T06:44:29.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:29.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:44:29.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481798 2024-06-03T06:44:29.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:29.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481798 2024-06-03T06:44:29.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-06-03T06:44:29.340 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.9 seq 330712481798 2024-06-03T06:44:29.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:44:29.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481798 -lt 330712481798 2024-06-03T06:44:29.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:44:29.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:44:29.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:44:29.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:44:29.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:44:29.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:44:29.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:44:29.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:44:29.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:44:29.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:44:29.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:44:30.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:44:30.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:44:30.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:44:30.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:44:30.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:44:30.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:44:30.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:44:30.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-03T06:44:30.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:44:30.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:44:30.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:44:30.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:44:30.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-03T06:44:30.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-03T06:44:30.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:44:30.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:44:30.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:44:30.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:44:30.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:44:30.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:44:31.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:44:00.897568+0000 2024-06-03T06:44:31.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:44:31.489 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:44:31.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:44:00.897568+0000 2024-06-03T06:44:31.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:44:31.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:44:00.897568+0000 2024-06-03T06:44:31.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:44:31.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:44:31.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:44:31.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:44:31.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:44:31.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:44:31.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:44:31.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:44:31.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:44:00.897568+0000 '>' 2024-06-03T06:44:00.897568+0000 2024-06-03T06:44:31.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:44:32.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:44:32.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:44:32.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:44:32.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:44:32.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:44:32.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:44:32.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:44:33.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:44:00.897568+0000 '>' 2024-06-03T06:44:00.897568+0000 2024-06-03T06:44:33.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:44:34.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:44:34.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:44:34.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:44:34.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:44:34.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:44:34.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:44:34.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:44:34.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:44:00.897568+0000 '>' 2024-06-03T06:44:00.897568+0000 2024-06-03T06:44:34.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:44:35.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:44:35.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:44:35.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:44:35.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:44:35.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:44:35.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:44:35.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:44:35.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:44:00.897568+0000 '>' 2024-06-03T06:44:00.897568+0000 2024-06-03T06:44:35.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:44:36.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:44:36.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:44:36.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:44:36.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:44:36.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:44:36.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:44:36.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:44:37.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:44:31.893527+0000 '>' 2024-06-03T06:44:00.897568+0000 2024-06-03T06:44:37.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:44:37.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:44:37.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:44:37.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:44:37.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-03T06:44:37.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:44:37.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:44:37.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:44:37.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:44:37.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-03T06:44:37.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:44:37.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-03T06:44:37.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:44:37.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:44:37.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:44:37.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:44:37.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:44:37.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:44:37.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:44:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:44:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:44:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-03T06:44:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:44:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:44:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-03T06:44:39.304 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:44:39.305 INFO:tasks.workunit.client.0.smithi039.stdout:hinfo_key 2024-06-03T06:44:39.305 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:44:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-03T06:44:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:44:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:44:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-03T06:44:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:44:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:44:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:44:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:44:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:44:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:44:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-03T06:44:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-03T06:44:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:44:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:44:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:44:39.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:44:39.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:44:39.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:44:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:44:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:44:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:44:39.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:44:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:44:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:44:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:44:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:44:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:44:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:44:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:44:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:44:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:44:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:44:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-03T06:44:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-03T06:44:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.9 2024-06-03T06:44:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:44:39.582 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-03T06:44:39.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-03T06:44:39.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:44:39.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:44:39.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:44:39.610 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:44:39.610+0000 7fbed3114640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:44:39.613 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:44:39.613+0000 7fbed3114640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:44:39.615 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:44:39.614+0000 7fbed3114640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:44:39.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-03T06:44:39.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:44:39.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-03T06:44:39.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:44:39.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:44:39.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:39.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:44:39.884 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:44:39.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:39.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:44:40.130 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:44:40.130+0000 7fbed3114640 -1 Falling back to public interface 2024-06-03T06:44:40.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:44:41.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:44:41.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:41.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:44:41.174 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:44:41.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:41.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:44:41.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:44:42.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:44:42.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:42.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:44:42.476 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:44:42.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:42.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:44:42.503 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:44:42.503+0000 7fbed3114640 -1 osd.9 85 log_to_monitors true 2024-06-03T06:44:42.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:44:43.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:44:43.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:43.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:44:43.808 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:44:43.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:43.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:44:44.244 INFO:tasks.workunit.client.0.smithi039.stderr:osd.9 up in weight 1 up_from 88 up_thru 0 down_at 85 last_clean_interval [77,85) [v2:127.0.0.1:6874/910342876,v1:127.0.0.1:6875/910342876] [v2:127.0.0.1:6876/910342876,v1:127.0.0.1:6877/910342876] exists,up 2a84bc6a-1558-4d3e-8c00-d333d070e6fc 2024-06-03T06:44:44.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:44:44.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:44:44.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:44:44.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:44:44.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:44:44.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:44:44.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:44:44.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:44:44.246 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:44:44.246 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:44:44.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:44:44.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:44:44.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:44:44.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:44:44.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:44:44.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:44:44.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:44:44.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:44:44.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:44:44.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:44:44.866 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:44:44.866 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:44:44.866 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:44:44.866 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-06-03T06:44:44.866 INFO:tasks.workunit.client.0.smithi039.stderr:5 2024-06-03T06:44:44.866 INFO:tasks.workunit.client.0.smithi039.stderr:6 2024-06-03T06:44:44.866 INFO:tasks.workunit.client.0.smithi039.stderr:7 2024-06-03T06:44:44.867 INFO:tasks.workunit.client.0.smithi039.stderr:8 2024-06-03T06:44:44.867 INFO:tasks.workunit.client.0.smithi039.stderr:9' 2024-06-03T06:44:44.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:44:44.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:44.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:44:44.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836565 2024-06-03T06:44:44.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836565 2024-06-03T06:44:44.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565' 2024-06-03T06:44:44.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:44.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:44:45.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673040 2024-06-03T06:44:45.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673040 2024-06-03T06:44:45.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040' 2024-06-03T06:44:45.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:45.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:44:45.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509514 2024-06-03T06:44:45.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509514 2024-06-03T06:44:45.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509514' 2024-06-03T06:44:45.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:45.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:44:45.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645327 2024-06-03T06:44:45.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645327 2024-06-03T06:44:45.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509514 3-309237645327' 2024-06-03T06:44:45.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:45.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:44:45.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182460 2024-06-03T06:44:45.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182460 2024-06-03T06:44:45.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509514 3-309237645327 4-107374182460' 2024-06-03T06:44:45.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:45.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:44:45.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318279 2024-06-03T06:44:45.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318279 2024-06-03T06:44:45.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509514 3-309237645327 4-107374182460 5-352187318279' 2024-06-03T06:44:45.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:45.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:44:45.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855407 2024-06-03T06:44:45.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855407 2024-06-03T06:44:45.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509514 3-309237645327 4-107374182460 5-352187318279 6-150323855407' 2024-06-03T06:44:45.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:45.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:44:45.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691882 2024-06-03T06:44:45.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691882 2024-06-03T06:44:45.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509514 3-309237645327 4-107374182460 5-352187318279 6-150323855407 7-171798691882' 2024-06-03T06:44:45.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:45.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:44:45.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528354 2024-06-03T06:44:45.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528354 2024-06-03T06:44:45.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509514 3-309237645327 4-107374182460 5-352187318279 6-150323855407 7-171798691882 8-193273528354' 2024-06-03T06:44:45.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:44:45.859 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:44:45.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-06-03T06:44:45.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-06-03T06:44:45.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509514 3-309237645327 4-107374182460 5-352187318279 6-150323855407 7-171798691882 8-193273528354 9-377957122050' 2024-06-03T06:44:45.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:45.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836565 2024-06-03T06:44:45.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:45.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:44:45.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836565 2024-06-03T06:44:45.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:45.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836565 2024-06-03T06:44:45.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-06-03T06:44:45.970 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836565 2024-06-03T06:44:45.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:44:46.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836565 -lt 21474836565 2024-06-03T06:44:46.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:46.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673040 2024-06-03T06:44:46.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:46.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:44:46.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673040 2024-06-03T06:44:46.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:46.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673040 2024-06-03T06:44:46.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-06-03T06:44:46.264 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949673040 2024-06-03T06:44:46.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:44:46.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673041 -lt 42949673040 2024-06-03T06:44:46.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:46.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509514 2024-06-03T06:44:46.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:46.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:44:46.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509514 2024-06-03T06:44:46.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:46.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509514 2024-06-03T06:44:46.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-06-03T06:44:46.557 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509514 2024-06-03T06:44:46.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:44:46.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509514 -lt 64424509514 2024-06-03T06:44:46.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:46.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645327 2024-06-03T06:44:46.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:46.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:44:46.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645327 2024-06-03T06:44:46.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:46.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645327 2024-06-03T06:44:46.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645327' 2024-06-03T06:44:46.858 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 309237645327 2024-06-03T06:44:46.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:44:47.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645328 -lt 309237645327 2024-06-03T06:44:47.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:47.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182460 2024-06-03T06:44:47.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:47.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:44:47.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182460 2024-06-03T06:44:47.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182460 2024-06-03T06:44:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182460' 2024-06-03T06:44:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.4 seq 107374182460 2024-06-03T06:44:47.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:44:47.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182460 -lt 107374182460 2024-06-03T06:44:47.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:47.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-352187318279 2024-06-03T06:44:47.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:47.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:44:47.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-352187318279 2024-06-03T06:44:47.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:47.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318279 2024-06-03T06:44:47.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 352187318279' 2024-06-03T06:44:47.443 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.5 seq 352187318279 2024-06-03T06:44:47.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:44:47.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318279 -lt 352187318279 2024-06-03T06:44:47.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:47.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855407 2024-06-03T06:44:47.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:47.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:44:47.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855407 2024-06-03T06:44:47.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:47.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855407 2024-06-03T06:44:47.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855407' 2024-06-03T06:44:47.740 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.6 seq 150323855407 2024-06-03T06:44:47.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:44:48.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855407 -lt 150323855407 2024-06-03T06:44:48.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:48.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:48.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691882 2024-06-03T06:44:48.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:44:48.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691882 2024-06-03T06:44:48.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:48.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691882 2024-06-03T06:44:48.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691882' 2024-06-03T06:44:48.033 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.7 seq 171798691882 2024-06-03T06:44:48.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:44:48.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691883 -lt 171798691882 2024-06-03T06:44:48.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:48.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528354 2024-06-03T06:44:48.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:48.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:44:48.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528354 2024-06-03T06:44:48.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:48.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528354 2024-06-03T06:44:48.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528354' 2024-06-03T06:44:48.413 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.8 seq 193273528354 2024-06-03T06:44:48.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:44:48.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528354 -lt 193273528354 2024-06-03T06:44:48.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:44:48.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-377957122050 2024-06-03T06:44:48.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:44:48.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:44:48.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-377957122050 2024-06-03T06:44:48.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:44:48.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-06-03T06:44:48.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 377957122050' 2024-06-03T06:44:48.707 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.9 seq 377957122050 2024-06-03T06:44:48.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:44:48.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122051 -lt 377957122050 2024-06-03T06:44:48.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:44:48.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:44:48.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:44:49.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:44:49.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:44:49.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:44:49.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:44:49.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:44:49.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:44:49.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:44:49.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:44:49.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:44:49.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:44:49.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:44:49.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:44:50.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-03T06:44:50.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-03T06:44:50.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:44:50.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-03T06:44:50.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:44:50.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:44:50.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:44:50.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:44:50.107 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:44:50.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:44:50.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:44:50.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:44:50.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:44:50.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:44:50.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:44:50.393 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:44:50.393 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:44:50.393 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:44:50.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:44:50.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:44:50.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:44:50.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:44:50.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:44:50.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-06-03T06:44:50.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-03T06:44:50.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-03T06:44:50.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-06-03T06:44:50.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-06-03T06:44:50.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-03T06:44:50.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:44:50.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:44:50.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:44:50.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 229260"' 2024-06-03T06:44:50.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 229260' 2024-06-03T06:44:50.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-06-03T06:44:50.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:44:50.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:44:50.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:44:50.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 229262"' 2024-06-03T06:44:50.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 229262' 2024-06-03T06:44:50.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/229263: /' 2024-06-03T06:44:50.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-03T06:44:50.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 229260 229262' 2024-06-03T06:44:50.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:44:50.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:44:50.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 229260 2024-06-03T06:44:50.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-06-03T06:44:50.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/229266: /' 2024-06-03T06:44:54.001 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:44:54.001 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:44:54.001 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-03T06:44:54.001 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:44:54.001 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-06-03T06:44:54.001 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:44:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:44:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-03T06:44:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:44:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-03T06:44:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:44:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:44:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:44:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:44:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:44:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:44:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-06-03T06:44:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:44:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:44:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-03T06:44:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:44:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:44:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-06-03T06:44:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:229266: remove 2#2:eb822e21:::SOMETHING:head# 2024-06-03T06:44:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-03T06:44:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:44:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:44:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-03T06:44:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:44:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:44:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:44:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:44:54.005 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:44:54.005 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:44:54.005 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-03T06:44:54.005 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-03T06:44:54.005 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:44:54.005 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:44:54.005 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:44:54.005 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:44:54.005 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:44:54.005 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:44:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:229266: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:44:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:229266: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:44:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:229266: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:44:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:44:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:44:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:44:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:44:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:44:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:44:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:44:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:44:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:44:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:44:54.007 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:44:54.007 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-03T06:44:54.007 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-03T06:44:54.007 INFO:tasks.workunit.client.0.smithi039.stderr:229266: start osd.9 2024-06-03T06:44:54.007 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:44:54.049 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:44:54.049 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:44:54.049 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-03T06:44:54.049 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:44:54.049 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:44:54.049 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:44:54.049 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:44:54.049 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-03T06:44:54.049 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:44:54.049 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-03T06:44:54.049 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:44:54.049 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:44:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:44:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:44:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:44:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:44:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:44:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:44:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:44:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-03T06:44:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:44:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:44:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-03T06:44:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:229263: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-03T06:44:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-03T06:44:54.051 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:44:54.051 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:44:54.051 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-03T06:44:54.051 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:44:54.051 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:44:54.051 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:44:54.051 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:44:54.052 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:44:54.052 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:44:54.052 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-03T06:44:54.052 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-03T06:44:54.052 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:44:54.053 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:44:54.053 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:44:54.053 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:44:54.053 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:44:54.053 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:44:54.053 INFO:tasks.workunit.client.0.smithi039.stderr:229263: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:44:54.053 INFO:tasks.workunit.client.0.smithi039.stderr:229263: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:44:54.053 INFO:tasks.workunit.client.0.smithi039.stderr:229263: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:44:54.053 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:44:54.053 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:44:54.053 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:44:54.053 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:44:54.053 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:44:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:44:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:44:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:44:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:44:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:44:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:44:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-03T06:44:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-03T06:44:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:229263: start osd.5 2024-06-03T06:44:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:44:58.769 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-03T06:44:58.769 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-03T06:44:58.769 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:44:58.769 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:44:58.769 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:44:58.770 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 2024-06-03T06:44:54.081+0000 7f2578d23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:44:58.770 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 2024-06-03T06:44:54.084+0000 7f2578d23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:44:58.770 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 2024-06-03T06:44:54.085+0000 7f2578d23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:44:58.770 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-03T06:44:58.770 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:44:58.770 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-03T06:44:58.770 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:44:58.770 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:44:58.770 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:58.770 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:44:58.771 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 0 2024-06-03T06:44:58.771 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:58.771 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:44:58.771 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 2024-06-03T06:44:54.600+0000 7f2578d23640 -1 Falling back to public interface 2024-06-03T06:44:58.771 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:44:58.771 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:44:58.771 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:58.771 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:44:58.771 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 1 2024-06-03T06:44:58.771 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:58.771 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:44:58.771 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:44:58.771 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:44:58.771 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:58.772 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:44:58.772 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 2 2024-06-03T06:44:58.772 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:58.772 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:44:58.772 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:44:58.772 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 2024-06-03T06:44:57.442+0000 7f2578d23640 -1 osd.5 90 log_to_monitors true 2024-06-03T06:44:58.772 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:44:58.772 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:58.772 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:44:58.772 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 3 2024-06-03T06:44:58.772 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:58.772 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:44:58.773 INFO:tasks.workunit.client.0.smithi039.stderr:229263: osd.5 up in weight 1 up_from 96 up_thru 82 down_at 91 last_229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-03T06:44:58.773 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-03T06:44:58.773 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:44:58.773 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:44:58.773 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:44:58.773 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 2024-06-03T06:44:54.035+0000 7fb44198e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:44:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 2024-06-03T06:44:54.038+0000 7fb44198e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:44:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 2024-06-03T06:44:54.039+0000 7fb44198e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:44:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-03T06:44:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:44:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-03T06:44:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:44:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:44:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:44:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 0 2024-06-03T06:44:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:44:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 2024-06-03T06:44:54.558+0000 7fb44198e640 -1 Falling back to public interface 2024-06-03T06:44:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:44:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:44:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:44:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 1 2024-06-03T06:44:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:44:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:44:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:44:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:44:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 2 2024-06-03T06:44:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:44:58.776 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 2024-06-03T06:44:57.246+0000 7fb44198e640 -1 osd.9 90 log_to_monitors true 2024-06-03T06:44:58.776 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:44:58.776 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:44:58.776 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:44:58.776 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:44:58.776 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 3 2024-06-03T06:44:58.776 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:44:58.776 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:44:59.997 INFO:tasks.workunit.client.0.smithi039.stderr:229266: osd.9 up in weight 1 up_from 95 up_thru 0 down_at 91 last_clean_interval [88,90) [v2:127.0.0.1:6842/2940452509,v1:127.0.0.1:6843/2940452509] [v2:127.0.0.1:6844/2940452509,v1:127.0.0.1:6845/2940452509] exists,up 2a84bc6a-1558-4d3e-8c00-d333d070e6fc 2024-06-03T06:44:59.998 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:44:59.998 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:44:59.998 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:44:59.998 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:44:59.998 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:44:59.998 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:44:59.998 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:44:59.998 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:44:59.998 INFO:tasks.workunit.client.0.smithi039.stderr:229266: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:44:59.998 INFO:tasks.workunit.client.0.smithi039.stderr:229266: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:44:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:44:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:44:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:44:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:44:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:44:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:44:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:44:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:44:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:44:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:44:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 1 2024-06-03T06:44:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 2 2024-06-03T06:44:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 3 2024-06-03T06:44:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 4 2024-06-03T06:45:00.000 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 5 2024-06-03T06:45:00.000 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 6 2024-06-03T06:45:00.000 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 7 2024-06-03T06:45:00.000 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 8 2024-06-03T06:45:00.000 INFO:tasks.workunit.client.0.smithi039.stderr:229266: 9' 2024-06-03T06:45:00.000 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:45:00.000 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:00.000 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:45:00.000 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836569 2024-06-03T06:45:00.000 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836569 2024-06-03T06:45:00.000 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569' 2024-06-03T06:45:00.000 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:00.000 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:45:00.000 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673045 2024-06-03T06:45:00.001 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673045 2024-06-03T06:45:00.001 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673045' 2024-06-03T06:45:00.001 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:00.001 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qaclean_interval [82,90) [v2:127.0.0.1:6874/310121281,v1:127.0.0.1:6875/310121281] [v2:127.0.0.1:6876/310121281,v1:127.0.0.1:6877/310121281] exists,up c849e020-3670-4066-b986-5db1f468a8d7 2024-06-03T06:45:00.001 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:45:00.001 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:45:00.001 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:45:00.001 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:45:00.001 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:45:00.001 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:45:00.001 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:45:00.001 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:45:00.001 INFO:tasks.workunit.client.0.smithi039.stderr:229263: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:45:00.001 INFO:tasks.workunit.client.0.smithi039.stderr:229263: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:45:00.002 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:45:00.002 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:45:00.002 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:45:00.002 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:45:00.002 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:45:00.002 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:45:00.002 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:45:00.002 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:45:00.002 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:45:00.002 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:45:00.002 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 1 2024-06-03T06:45:00.002 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 2 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 3 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 4 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 5 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 6 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 7 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 8 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: 9' 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836570 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836570 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570' 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:45:00.003 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673044 2024-06-03T06:45:00.004 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673044 2024-06-03T06:45:00.004 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673044' 2024-06-03T06:45:00.004 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:01.237 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:45:01.237 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509518 2024-06-03T06:45:01.238 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509518 2024-06-03T06:45:01.238 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673044 2-64424509518' 2024-06-03T06:45:01.238 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:01.238 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:45:01.238 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645332 2024-06-03T06:45:01.238 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645332 2024-06-03T06:45:01.238 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673044 2-64424509518 3-309237645332' 2024-06-03T06:45:01.238 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:01.238 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:45:01.238 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182465 2024-06-03T06:45:01.238 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182465 2024-06-03T06:45:01.238 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673044 2-64424509518 3-309237645332 4-107374182465' 2024-06-03T06:45:01.239 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:01.239 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:45:01.239 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-06-03T06:45:01.239 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-06-03T06:45:01.239 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673044 2-64424509518 3-309237645332 4-107374182465 5-412316860418' 2024-06-03T06:45:01.239 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:01.239 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:45:01.239 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855412 2024-06-03T06:45:01.239 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855412 2024-06-03T06:45:01.239 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673044 2-64424509518 3-309237645332 4-107374182465 5-412316860418 6-150323855412' 2024-06-03T06:45:01.239 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:01.240 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:45:01.240 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691887 2024-06-03T06:45:01.240 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691887 2024-06-03T06:45:01.240 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673044 2-64424509518 3-309237645332 4-107374182465 5-412316860418 6-150323855412 7-171798691887' 2024-06-03T06:45:01.240 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:01.240 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:45:01.240 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.cl/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:45:01.240 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509519 2024-06-03T06:45:01.240 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509519 2024-06-03T06:45:01.240 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509519' 2024-06-03T06:45:01.240 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:01.240 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:45:01.241 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645331 2024-06-03T06:45:01.241 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645331 2024-06-03T06:45:01.241 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509519 3-309237645331' 2024-06-03T06:45:01.241 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:01.241 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:45:01.241 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182464 2024-06-03T06:45:01.241 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182464 2024-06-03T06:45:01.241 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509519 3-309237645331 4-107374182464' 2024-06-03T06:45:01.241 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:01.241 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:45:01.242 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860419 2024-06-03T06:45:01.242 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860419 2024-06-03T06:45:01.242 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509519 3-309237645331 4-107374182464 5-412316860419' 2024-06-03T06:45:01.242 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:01.242 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:45:01.242 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855411 2024-06-03T06:45:01.242 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855411 2024-06-03T06:45:01.242 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509519 3-309237645331 4-107374182464 5-412316860419 6-150323855411' 2024-06-03T06:45:01.242 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:01.242 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:45:01.242 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691886 2024-06-03T06:45:01.242 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691886 2024-06-03T06:45:01.243 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509519 3-309237645331 4-107374182464 5-412316860419 6-150323855411 7-171798691886' 2024-06-03T06:45:01.243 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:01.243 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:45:01.980 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528359 2024-06-03T06:45:01.980 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528359 2024-06-03T06:45:01.980 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509519 3-309237645331 4-107374182464 5-412316860419 6-150323855411 7-171798691886 8-193273528359' 2024-06-03T06:45:01.980 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:01.980 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:45:01.980 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893122 2024-06-03T06:45:01.980 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893122 2024-06-03T06:45:01.980 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509519 3-309237645331 4-107374182464 5-412316860419 6-150323855411 7-171798691886 8-193273528359 9-408021893122' 2024-06-03T06:45:01.980 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:01.980 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836569 2024-06-03T06:45:01.980 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:01.981 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:45:01.981 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836569 2024-06-03T06:45:01.981 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:01.981 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836569 2024-06-03T06:45:01.981 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-06-03T06:45:01.981 INFO:tasks.workunit.client.0.smithi039.stderr:229266: waiting osd.0 seq 21474836569 2024-06-03T06:45:01.981 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:45:01.981 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836570 -lt 21474836569 2024-06-03T06:45:01.981 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:01.981 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673045 2024-06-03T06:45:01.981 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:01.981 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:45:01.981 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673045 2024-06-03T06:45:01.981 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:01.981 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673045 2024-06-03T06:45:01.982 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-06-03T06:45:01.982 INFO:tasks.workunit.client.0.smithi039.stderr:229266: waiting osd.1 seq 42949673045 2024-06-03T06:45:01.982 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:45:01.982 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673045 -lt 42949673045 2024-06-03T06:45:01.982 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:01.982 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509519 2024-06-03T06:45:01.982 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:01.982 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:45:01.986 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.cient.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528358 2024-06-03T06:45:01.986 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528358 2024-06-03T06:45:01.986 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673044 2-64424509518 3-309237645332 4-107374182465 5-412316860418 6-150323855412 7-171798691887 8-193273528358' 2024-06-03T06:45:01.986 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:01.986 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:45:01.986 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893123 2024-06-03T06:45:01.986 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893123 2024-06-03T06:45:01.986 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673044 2-64424509518 3-309237645332 4-107374182465 5-412316860418 6-150323855412 7-171798691887 8-193273528358 9-408021893123' 2024-06-03T06:45:01.987 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:01.987 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836570 2024-06-03T06:45:01.987 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:01.987 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:45:01.987 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836570 2024-06-03T06:45:01.987 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:01.987 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836570 2024-06-03T06:45:01.987 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2024-06-03T06:45:01.987 INFO:tasks.workunit.client.0.smithi039.stderr:229263: waiting osd.0 seq 21474836570 2024-06-03T06:45:01.987 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:45:01.987 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836570 -lt 21474836570 2024-06-03T06:45:01.987 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:01.988 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673044 2024-06-03T06:45:01.988 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:01.988 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:45:01.988 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673044 2024-06-03T06:45:01.988 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:01.988 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673044 2024-06-03T06:45:01.988 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673044' 2024-06-03T06:45:01.988 INFO:tasks.workunit.client.0.smithi039.stderr:229263: waiting osd.1 seq 42949673044 2024-06-03T06:45:01.988 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:45:01.988 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673045 -lt 42949673044 2024-06-03T06:45:01.988 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:01.988 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509518 2024-06-03T06:45:01.988 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:01.989 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:45:03.006 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509518 2024-06-03T06:45:03.007 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:03.007 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509518 2024-06-03T06:45:03.007 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2024-06-03T06:45:03.007 INFO:tasks.workunit.client.0.smithi039.stderr:229263: waiting osd.2 seq 64424509518 2024-06-03T06:45:03.007 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:45:03.007 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509519 -lt 64424509518 2024-06-03T06:45:03.007 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:03.007 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645332 2024-06-03T06:45:03.007 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:03.007 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:45:03.007 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645332 2024-06-03T06:45:03.007 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645332 2024-06-03T06:45:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645332' 2024-06-03T06:45:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:229263: waiting osd.3 seq 309237645332 2024-06-03T06:45:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:45:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645333 -lt 309237645332 2024-06-03T06:45:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182465 2024-06-03T06:45:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:45:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182465 2024-06-03T06:45:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182465 2024-06-03T06:45:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182465' 2024-06-03T06:45:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:229263: waiting osd.4 seq 107374182465 2024-06-03T06:45:03.009 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:45:03.009 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182465 -lt 107374182465 2024-06-03T06:45:03.009 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:03.009 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-412316860418 2024-06-03T06:45:03.009 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:03.009 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:45:03.009 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-412316860418 2024-06-03T06:45:03.009 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:03.009 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-06-03T06:45:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting lient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509519 2024-06-03T06:45:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509519 2024-06-03T06:45:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509519' 2024-06-03T06:45:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:229266: waiting osd.2 seq 64424509519 2024-06-03T06:45:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:45:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509519 -lt 64424509519 2024-06-03T06:45:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645331 2024-06-03T06:45:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:45:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645331 2024-06-03T06:45:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645331 2024-06-03T06:45:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645331' 2024-06-03T06:45:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:229266: waiting osd.3 seq 309237645331 2024-06-03T06:45:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:45:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645333 -lt 309237645331 2024-06-03T06:45:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182464 2024-06-03T06:45:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:45:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182464 2024-06-03T06:45:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182464 2024-06-03T06:45:03.016 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182464' 2024-06-03T06:45:03.016 INFO:tasks.workunit.client.0.smithi039.stderr:229266: waiting osd.4 seq 107374182464 2024-06-03T06:45:03.016 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:45:03.016 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182465 -lt 107374182464 2024-06-03T06:45:03.016 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:03.016 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-412316860419 2024-06-03T06:45:03.016 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:03.016 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:45:03.016 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-412316860419 2024-06-03T06:45:03.016 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:03.016 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860419 2024-06-03T06:45:04.305 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitiosd.5 seq 412316860418' 2024-06-03T06:45:04.305 INFO:tasks.workunit.client.0.smithi039.stderr:229263: waiting osd.5 seq 412316860418 2024-06-03T06:45:04.306 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:45:04.306 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860419 -lt 412316860418 2024-06-03T06:45:04.306 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:04.306 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855412 2024-06-03T06:45:04.306 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:04.306 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:45:04.306 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855412 2024-06-03T06:45:04.306 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:04.306 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855412 2024-06-03T06:45:04.306 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855412' 2024-06-03T06:45:04.306 INFO:tasks.workunit.client.0.smithi039.stderr:229263: waiting osd.6 seq 150323855412 2024-06-03T06:45:04.306 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:45:04.307 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855412 -lt 150323855412 2024-06-03T06:45:04.307 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:04.307 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691887 2024-06-03T06:45:04.307 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:04.307 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:45:04.307 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691887 2024-06-03T06:45:04.307 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:04.307 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691887 2024-06-03T06:45:04.307 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691887' 2024-06-03T06:45:04.307 INFO:tasks.workunit.client.0.smithi039.stderr:229263: waiting osd.7 seq 171798691887 2024-06-03T06:45:04.307 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:45:04.307 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691887 -lt 171798691887 2024-06-03T06:45:04.307 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:04.307 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528358 2024-06-03T06:45:04.307 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:04.308 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:45:04.308 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528358 2024-06-03T06:45:04.308 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:04.308 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528358 2024-06-03T06:45:04.308 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2024-06-03T06:45:04.308 INFO:tasks.workunit.client.0.smithi039.stderr:229263: waiting osd.8 seq 193273528358 2024-06-03T06:45:04.308 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:45:04.308 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528359 -lt 193273528358 2024-06-03T06:45:04.380 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stng osd.5 seq 412316860419' 2024-06-03T06:45:04.381 INFO:tasks.workunit.client.0.smithi039.stderr:229266: waiting osd.5 seq 412316860419 2024-06-03T06:45:04.381 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:45:04.381 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860419 -lt 412316860419 2024-06-03T06:45:04.381 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:04.381 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855411 2024-06-03T06:45:04.381 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:04.381 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:45:04.381 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855411 2024-06-03T06:45:04.381 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:04.381 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855411 2024-06-03T06:45:04.381 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855411' 2024-06-03T06:45:04.381 INFO:tasks.workunit.client.0.smithi039.stderr:229266: waiting osd.6 seq 150323855411 2024-06-03T06:45:04.382 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:45:04.382 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855412 -lt 150323855411 2024-06-03T06:45:04.382 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:04.382 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691886 2024-06-03T06:45:04.382 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:04.382 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:45:04.382 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691886 2024-06-03T06:45:04.382 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:04.382 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691886 2024-06-03T06:45:04.382 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691886' 2024-06-03T06:45:04.382 INFO:tasks.workunit.client.0.smithi039.stderr:229266: waiting osd.7 seq 171798691886 2024-06-03T06:45:04.382 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:45:04.382 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691887 -lt 171798691886 2024-06-03T06:45:04.383 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:04.383 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528359 2024-06-03T06:45:04.383 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:04.383 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:45:04.383 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528359 2024-06-03T06:45:04.383 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:04.383 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528359 2024-06-03T06:45:04.383 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528359' 2024-06-03T06:45:04.383 INFO:tasks.workunit.client.0.smithi039.stderr:229266: waiting osd.8 seq 193273528359 2024-06-03T06:45:04.383 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:45:04.383 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528359 -lt 193273528359 2024-06-03T06:45:05.714 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pgats: for s in $seqs 2024-06-03T06:45:05.714 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-408021893123 2024-06-03T06:45:05.714 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:05.714 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:45:05.714 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-408021893123 2024-06-03T06:45:05.714 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:05.714 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893123 2024-06-03T06:45:05.714 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 408021893123' 2024-06-03T06:45:05.714 INFO:tasks.workunit.client.0.smithi039.stderr:229263: waiting osd.9 seq 408021893123 2024-06-03T06:45:05.714 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:45:05.714 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893123 2024-06-03T06:45:05.714 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:45:05.715 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:05.715 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:05.715 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:45:05.715 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:45:05.715 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:45:05.715 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:45:05.715 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:45:05.715 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:45:05.715 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:45:05.715 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:45:05.715 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:45:05.715 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:45:05.715 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:05.716 INFO:tasks.workunit.client.0.smithi039.stderr:229263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:05.716 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:45:05.716 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:45:05.716 INFO:tasks.workunit.client.0.smithi039.stderr:229263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:45:05.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:45:05.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:45:05.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 229262 2024-06-03T06:45:05.745 INFO:tasks.workunit.client.0.smithi039.stderr:_stats: for s in $seqs 2024-06-03T06:45:05.745 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-408021893122 2024-06-03T06:45:05.745 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:05.745 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:45:05.745 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-408021893122 2024-06-03T06:45:05.746 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:05.746 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893122 2024-06-03T06:45:05.746 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 408021893122' 2024-06-03T06:45:05.746 INFO:tasks.workunit.client.0.smithi039.stderr:229266: waiting osd.9 seq 408021893122 2024-06-03T06:45:05.746 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:45:05.746 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893122 2024-06-03T06:45:05.746 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:45:05.746 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:05.746 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:05.746 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:45:05.746 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:45:05.746 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:45:05.746 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:45:05.747 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:45:05.747 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:45:05.747 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:45:05.747 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:45:05.747 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:45:05.747 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:45:05.747 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:05.747 INFO:tasks.workunit.client.0.smithi039.stderr:229266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:05.747 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:45:05.747 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:45:05.747 INFO:tasks.workunit.client.0.smithi039.stderr:229266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:45:05.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:45:05.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:45:05.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:45:05.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:45:05.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-03T06:45:05.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:45:05.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-03T06:45:05.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:45:05.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:45:05.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:45:05.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:45:06.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-03T06:45:06.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-03T06:45:06.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:45:06.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:45:06.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:45:06.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:45:06.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:45:06.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:45:06.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:44:31.893527+0000 2024-06-03T06:45:06.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:45:06.641 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:45:06.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:44:31.893527+0000 2024-06-03T06:45:06.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:45:06.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:44:31.893527+0000 2024-06-03T06:45:06.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:45:06.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:45:06.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:45:06.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:45:06.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:45:06.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:45:06.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:45:06.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:45:06.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:44:31.893527+0000 '>' 2024-06-03T06:44:31.893527+0000 2024-06-03T06:45:06.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:45:07.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:45:07.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:45:07.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:45:07.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:45:07.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:45:07.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:45:07.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:45:08.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:44:31.893527+0000 '>' 2024-06-03T06:44:31.893527+0000 2024-06-03T06:45:08.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:45:09.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:45:09.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:45:09.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:45:09.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:45:09.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:45:09.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:45:09.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:45:09.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:44:31.893527+0000 '>' 2024-06-03T06:44:31.893527+0000 2024-06-03T06:45:09.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:45:10.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:45:10.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:45:10.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:45:10.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:45:10.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:45:10.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:45:10.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:45:10.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:44:31.893527+0000 '>' 2024-06-03T06:44:31.893527+0000 2024-06-03T06:45:10.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:45:11.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:45:11.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:45:11.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:45:11.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:45:11.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:45:11.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:45:11.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:45:12.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:45:07.692090+0000 '>' 2024-06-03T06:44:31.893527+0000 2024-06-03T06:45:12.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:45:12.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-03T06:45:12.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:45:12.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:45:12.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:45:12.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 232666"' 2024-06-03T06:45:12.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 232666' 2024-06-03T06:45:12.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:45:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:45:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:45:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 232668"' 2024-06-03T06:45:12.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 232668' 2024-06-03T06:45:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-03T06:45:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 232666 232668' 2024-06-03T06:45:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:45:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:45:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:45:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 232666 2024-06-03T06:45:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/232669: /' 2024-06-03T06:45:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:45:12.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/232671: /' 2024-06-03T06:45:15.045 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:45:15.046 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:45:15.046 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-03T06:45:15.046 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:45:15.046 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:45:15.046 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:45:15.046 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:45:15.046 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-03T06:45:15.046 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:45:15.046 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-03T06:45:15.046 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:45:15.046 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:45:15.046 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:45:15.047 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:45:15.047 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:45:15.047 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:45:15.047 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:45:15.047 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:45:15.047 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:45:15.047 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-03T06:45:15.047 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:45:15.047 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:45:15.047 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-03T06:45:15.047 INFO:tasks.workunit.client.0.smithi039.stderr:232671: _ 2024-06-03T06:45:15.047 INFO:tasks.workunit.client.0.smithi039.stderr:232671: hinfo_key 2024-06-03T06:45:15.047 INFO:tasks.workunit.client.0.smithi039.stderr:232671: snapset 2024-06-03T06:45:15.048 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-03T06:45:15.048 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:45:15.048 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:45:15.048 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-03T06:45:15.048 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:45:15.048 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:45:15.048 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:45:15.048 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:45:15.049 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:45:15.049 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:45:15.049 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-03T06:45:15.049 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-03T06:45:15.049 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:45:15.049 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:45:15.049 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:45:15.049 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:45:15.049 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:45:15.049 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:45:15.049 INFO:tasks.workunit.client.0.smithi039.stderr:232671: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:45:15.050 INFO:tasks.workunit.client.0.smithi039.stderr:232671: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:45:15.050 INFO:tasks.workunit.client.0.smithi039.stderr:232671: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:45:15.050 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:45:15.050 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:45:15.050 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:45:15.050 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:45:15.050 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:45:15.050 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:45:15.050 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:45:15.050 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:45:15.050 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:45:15.051 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:45:15.051 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:45:15.051 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-03T06:45:15.051 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-03T06:45:15.051 INFO:tasks.workunit.client.0.smithi039.stderr:232671: start osd.9 2024-06-03T06:45:15.138 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:45:15.139 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:45:15.139 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-03T06:45:15.139 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:45:15.139 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:45:15.139 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:45:15.139 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:45:15.139 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-03T06:45:15.139 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:45:15.139 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-03T06:45:15.139 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:45:15.139 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:45:15.139 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:45:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:45:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:45:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:45:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:45:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:45:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:45:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-03T06:45:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:45:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:45:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-06-03T06:45:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:232669: _ 2024-06-03T06:45:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:232669: hinfo_key 2024-06-03T06:45:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:232669: snapset 2024-06-03T06:45:15.140 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-03T06:45:15.141 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:45:15.141 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:45:15.141 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-03T06:45:15.141 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:45:15.141 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:45:15.141 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:45:15.142 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:45:15.142 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:45:15.142 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:45:15.142 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-03T06:45:15.142 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-03T06:45:15.142 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:45:15.142 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:45:15.142 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:45:15.142 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:45:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:45:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:45:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:232669: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:45:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:232669: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:45:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:232669: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:45:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:45:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:45:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:45:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:45:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:45:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:45:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:45:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:45:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:45:15.144 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:45:15.144 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:45:15.144 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-03T06:45:15.144 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-03T06:45:15.144 INFO:tasks.workunit.client.0.smithi039.stderr:232669: start osd.5 2024-06-03T06:45:19.816 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:45:19.816 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-03T06:45:19.816 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-03T06:45:19.816 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:45:19.816 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:45:19.816 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:45:19.816 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 2024-06-03T06:45:15.079+0000 7f3576817640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:19.816 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 2024-06-03T06:45:15.081+0000 7f3576817640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:19.816 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 2024-06-03T06:45:15.083+0000 7f3576817640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:19.816 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-03T06:45:19.816 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:45:19.816 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-03T06:45:19.817 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:45:19.817 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:45:19.817 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:19.817 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:45:19.817 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 0 2024-06-03T06:45:19.817 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:19.817 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:45:19.817 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 2024-06-03T06:45:15.604+0000 7f3576817640 -1 Falling back to public interface 2024-06-03T06:45:19.817 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:19.817 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:19.817 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:19.818 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:45:19.818 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 1 2024-06-03T06:45:19.818 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:19.818 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:45:19.818 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:19.818 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:19.818 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:19.818 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:45:19.818 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 2 2024-06-03T06:45:19.818 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:19.818 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:45:19.819 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 2024-06-03T06:45:18.139+0000 7f3576817640 -1 osd.9 98 log_to_monitors true 2024-06-03T06:45:19.819 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:19.819 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:19.819 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:19.819 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:45:19.819 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 3 2024-06-03T06:45:19.819 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:19.819 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:45:19.819 INFO:tasks.workunit.client.0.smithi039.stderr:232671: osd.9 up in weight 1 up_from 103 up_thru 0 down_aovery_ops 2024-06-03T06:45:19.819 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-03T06:45:19.819 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-03T06:45:19.819 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:45:19.819 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:45:19.820 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:45:19.820 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 2024-06-03T06:45:15.170+0000 7f5d39bb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:19.820 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 2024-06-03T06:45:15.172+0000 7f5d39bb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:19.820 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 2024-06-03T06:45:15.173+0000 7f5d39bb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:19.820 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-03T06:45:19.820 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:45:19.820 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-03T06:45:19.820 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:45:19.820 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:45:19.820 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:19.820 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:45:19.820 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 0 2024-06-03T06:45:19.820 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:19.820 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:45:19.820 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 2024-06-03T06:45:15.691+0000 7f5d39bb6640 -1 Falling back to public interface 2024-06-03T06:45:19.821 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:19.821 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:19.821 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:19.821 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:45:19.821 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 1 2024-06-03T06:45:19.821 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:19.821 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:45:19.821 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:19.821 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:19.821 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:19.821 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:45:19.821 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 2 2024-06-03T06:45:19.821 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:19.821 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:45:19.822 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:19.822 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 2024-06-03T06:45:18.469+0000 7f5d39bb6640 -1 osd.5 98 log_to_monitors true 2024-06-03T06:45:19.822 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:19.822 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:19.822 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:45:19.822 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 3 2024-06-03T06:45:19.822 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:19.822 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:45:20.901 INFO:tasks.workunit.client.0.smithi039.stderr:232669: osd.5 up in weight 1 up_from 104 up_thru 96 down_at 99 last_clean_interval [96,98) [v2:127.0.0.1:6874/4255895709,v1:127.0.0.1:6875/4255895709] [v2:127.0.0.1:6876/4255895709,v1:127.0.0.1:6877/4255895709] exists,up c849e020-3670-4066-b986-5db1f468a8d7 2024-06-03T06:45:20.901 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:45:20.901 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:45:20.902 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:45:20.902 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:45:20.902 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:45:20.902 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:45:20.902 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:45:20.902 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:45:20.902 INFO:tasks.workunit.client.0.smithi039.stderr:232669: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:45:20.902 INFO:tasks.workunit.client.0.smithi039.stderr:232669: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:45:20.902 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:45:20.902 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:45:20.902 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:45:20.902 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:45:20.903 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:45:20.903 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:45:20.903 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:45:20.903 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:45:20.903 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:45:20.903 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:45:20.903 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 1 2024-06-03T06:45:20.903 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 2 2024-06-03T06:45:20.903 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 3 2024-06-03T06:45:20.903 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 4 2024-06-03T06:45:20.903 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 5 2024-06-03T06:45:20.903 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 6 2024-06-03T06:45:20.903 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 7 2024-06-03T06:45:20.904 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 8 2024-06-03T06:45:20.904 INFO:tasks.workunit.client.0.smithi039.stderr:232669: 9' 2024-06-03T06:45:20.904 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:45:20.904 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:20.904 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:45:20.904 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836575 2024-06-03T06:45:20.904 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836575 2024-06-03T06:45:20.904 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575' 2024-06-03T06:45:20.904 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:20.904 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:45:20.904 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673051 2024-06-03T06:45:20.904 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673051 2024-06-03T06:45:20.904 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673051' 2024-06-03T06:45:20.904 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:20.905 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clonet 99 last_clean_interval [95,98) [v2:127.0.0.1:6842/1173694271,v1:127.0.0.1:6843/1173694271] [v2:127.0.0.1:6844/1173694271,v1:127.0.0.1:6845/1173694271] exists,up 2a84bc6a-1558-4d3e-8c00-d333d070e6fc 2024-06-03T06:45:20.905 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:45:20.905 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:45:20.905 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:45:20.905 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:45:20.905 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:45:20.905 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:45:20.905 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:45:20.905 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:45:20.905 INFO:tasks.workunit.client.0.smithi039.stderr:232671: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:45:20.905 INFO:tasks.workunit.client.0.smithi039.stderr:232671: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:45:20.905 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:45:20.905 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:45:20.906 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:45:20.906 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:45:20.906 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:45:20.906 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:45:20.906 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:45:20.906 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:45:20.906 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:45:20.906 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:45:20.906 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 1 2024-06-03T06:45:20.906 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 2 2024-06-03T06:45:20.906 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 3 2024-06-03T06:45:20.906 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 4 2024-06-03T06:45:20.906 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 5 2024-06-03T06:45:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 6 2024-06-03T06:45:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 7 2024-06-03T06:45:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 8 2024-06-03T06:45:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:232671: 9' 2024-06-03T06:45:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:45:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:45:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836576 2024-06-03T06:45:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836576 2024-06-03T06:45:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576' 2024-06-03T06:45:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:45:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673052 2024-06-03T06:45:20.908 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673052 2024-06-03T06:45:20.908 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052' 2024-06-03T06:45:20.908 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:21.955 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:45:21.955 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509524 2024-06-03T06:45:21.955 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509524 2024-06-03T06:45:21.955 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509524' 2024-06-03T06:45:21.955 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:21.955 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:45:21.955 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645339 2024-06-03T06:45:21.955 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645339 2024-06-03T06:45:21.955 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509524 3-309237645339' 2024-06-03T06:45:21.956 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:21.956 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:45:21.956 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182471 2024-06-03T06:45:21.956 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182471 2024-06-03T06:45:21.956 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509524 3-309237645339 4-107374182471' 2024-06-03T06:45:21.956 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:21.956 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:45:21.956 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598787 2024-06-03T06:45:21.956 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598787 2024-06-03T06:45:21.956 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509524 3-309237645339 4-107374182471 5-446676598787' 2024-06-03T06:45:21.956 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:21.956 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:45:21.956 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855417 2024-06-03T06:45:21.957 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855417 2024-06-03T06:45:21.957 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509524 3-309237645339 4-107374182471 5-446676598787 6-150323855417' 2024-06-03T06:45:21.957 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:21.957 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:45:21.957 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691892 2024-06-03T06:45:21.957 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691892 2024-06-03T06:45:21.957 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509524 3-309237645339 4-107374182471 5-446676598787 6-150323855417 7-171798691892' 2024-06-03T06:45:21.957 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:21.957 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:45:21.957 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cep.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:45:21.957 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509525 2024-06-03T06:45:21.957 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509525 2024-06-03T06:45:21.957 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673051 2-64424509525' 2024-06-03T06:45:21.957 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:21.958 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:45:21.958 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645338 2024-06-03T06:45:21.958 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645338 2024-06-03T06:45:21.958 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673051 2-64424509525 3-309237645338' 2024-06-03T06:45:21.958 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:21.958 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:45:21.958 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182470 2024-06-03T06:45:21.958 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182470 2024-06-03T06:45:21.958 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673051 2-64424509525 3-309237645338 4-107374182470' 2024-06-03T06:45:21.958 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:21.958 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:45:21.958 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598786 2024-06-03T06:45:21.959 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598786 2024-06-03T06:45:21.959 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673051 2-64424509525 3-309237645338 4-107374182470 5-446676598786' 2024-06-03T06:45:21.959 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:21.959 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:45:21.959 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855418 2024-06-03T06:45:21.959 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855418 2024-06-03T06:45:21.959 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673051 2-64424509525 3-309237645338 4-107374182470 5-446676598786 6-150323855418' 2024-06-03T06:45:21.959 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:21.959 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:45:21.959 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691893 2024-06-03T06:45:21.959 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691893 2024-06-03T06:45:21.959 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673051 2-64424509525 3-309237645338 4-107374182470 5-446676598786 6-150323855418 7-171798691893' 2024-06-03T06:45:21.959 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:21.959 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:45:22.772 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cehtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528365 2024-06-03T06:45:22.772 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528365 2024-06-03T06:45:22.772 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509524 3-309237645339 4-107374182471 5-446676598787 6-150323855417 7-171798691892 8-193273528365' 2024-06-03T06:45:22.772 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:22.772 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:45:22.773 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631490 2024-06-03T06:45:22.773 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631490 2024-06-03T06:45:22.773 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509524 3-309237645339 4-107374182471 5-446676598787 6-150323855417 7-171798691892 8-193273528365 9-442381631490' 2024-06-03T06:45:22.773 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:22.773 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836576 2024-06-03T06:45:22.773 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:22.773 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:45:22.773 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836576 2024-06-03T06:45:22.773 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:22.773 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836576 2024-06-03T06:45:22.773 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836576' 2024-06-03T06:45:22.773 INFO:tasks.workunit.client.0.smithi039.stderr:232671: waiting osd.0 seq 21474836576 2024-06-03T06:45:22.774 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:45:22.774 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836576 2024-06-03T06:45:22.774 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:22.774 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673052 2024-06-03T06:45:22.774 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:22.774 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:45:22.774 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673052 2024-06-03T06:45:22.774 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:22.774 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673052 2024-06-03T06:45:22.774 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-06-03T06:45:22.775 INFO:tasks.workunit.client.0.smithi039.stderr:232671: waiting osd.1 seq 42949673052 2024-06-03T06:45:22.775 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:45:22.775 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673052 2024-06-03T06:45:22.775 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:22.775 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509524 2024-06-03T06:45:22.775 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:22.775 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:45:22.775 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528364 2024-06-03T06:45:22.776 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528364 2024-06-03T06:45:22.776 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673051 2-64424509525 3-309237645338 4-107374182470 5-446676598786 6-150323855418 7-171798691893 8-193273528364' 2024-06-03T06:45:22.776 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:22.776 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:45:22.776 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631491 2024-06-03T06:45:22.776 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631491 2024-06-03T06:45:22.776 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673051 2-64424509525 3-309237645338 4-107374182470 5-446676598786 6-150323855418 7-171798691893 8-193273528364 9-442381631491' 2024-06-03T06:45:22.776 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:22.776 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836575 2024-06-03T06:45:22.776 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:22.776 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:45:22.776 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836575 2024-06-03T06:45:22.777 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:22.777 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836575 2024-06-03T06:45:22.777 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2024-06-03T06:45:22.777 INFO:tasks.workunit.client.0.smithi039.stderr:232669: waiting osd.0 seq 21474836575 2024-06-03T06:45:22.777 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:45:22.777 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836575 2024-06-03T06:45:22.777 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:22.777 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673051 2024-06-03T06:45:22.777 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:22.777 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:45:22.777 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673051 2024-06-03T06:45:22.777 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:22.777 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673051 2024-06-03T06:45:22.778 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-06-03T06:45:22.778 INFO:tasks.workunit.client.0.smithi039.stderr:232669: waiting osd.1 seq 42949673051 2024-06-03T06:45:22.778 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:45:22.778 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673051 2024-06-03T06:45:22.778 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:22.778 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509525 2024-06-03T06:45:22.778 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:22.778 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:45:23.697 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509524 2024-06-03T06:45:23.698 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:23.698 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509524 2024-06-03T06:45:23.698 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-06-03T06:45:23.698 INFO:tasks.workunit.client.0.smithi039.stderr:232671: waiting osd.2 seq 64424509524 2024-06-03T06:45:23.698 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:45:23.698 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509525 -lt 64424509524 2024-06-03T06:45:23.698 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:23.698 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645339 2024-06-03T06:45:23.698 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:23.698 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:45:23.698 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645339 2024-06-03T06:45:23.699 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:23.699 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645339 2024-06-03T06:45:23.699 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645339' 2024-06-03T06:45:23.699 INFO:tasks.workunit.client.0.smithi039.stderr:232671: waiting osd.3 seq 309237645339 2024-06-03T06:45:23.699 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:45:23.699 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645339 -lt 309237645339 2024-06-03T06:45:23.699 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:23.699 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182471 2024-06-03T06:45:23.699 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:23.699 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:45:23.699 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182471 2024-06-03T06:45:23.699 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:23.699 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182471 2024-06-03T06:45:23.699 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182471' 2024-06-03T06:45:23.700 INFO:tasks.workunit.client.0.smithi039.stderr:232671: waiting osd.4 seq 107374182471 2024-06-03T06:45:23.700 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:45:23.700 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182471 -lt 107374182471 2024-06-03T06:45:23.700 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:23.700 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-446676598787 2024-06-03T06:45:23.700 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:23.700 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:45:23.700 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-446676598787 2024-06-03T06:45:23.700 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:23.700 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598787 2024-06-03T06:45:23.802 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: test/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509525 2024-06-03T06:45:23.802 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:23.802 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509525 2024-06-03T06:45:23.802 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-06-03T06:45:23.802 INFO:tasks.workunit.client.0.smithi039.stderr:232669: waiting osd.2 seq 64424509525 2024-06-03T06:45:23.802 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:45:23.802 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509525 -lt 64424509525 2024-06-03T06:45:23.803 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:23.803 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645338 2024-06-03T06:45:23.803 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:23.803 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:45:23.803 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645338 2024-06-03T06:45:23.803 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:23.803 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645338 2024-06-03T06:45:23.803 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645338' 2024-06-03T06:45:23.803 INFO:tasks.workunit.client.0.smithi039.stderr:232669: waiting osd.3 seq 309237645338 2024-06-03T06:45:23.803 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:45:23.803 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645339 -lt 309237645338 2024-06-03T06:45:23.803 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:23.804 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182470 2024-06-03T06:45:23.804 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:23.804 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:45:23.804 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182470 2024-06-03T06:45:23.804 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:23.804 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182470 2024-06-03T06:45:23.804 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182470' 2024-06-03T06:45:23.804 INFO:tasks.workunit.client.0.smithi039.stderr:232669: waiting osd.4 seq 107374182470 2024-06-03T06:45:23.804 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:45:23.804 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182471 -lt 107374182470 2024-06-03T06:45:23.804 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:23.804 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-446676598786 2024-06-03T06:45:23.804 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:23.805 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:45:23.805 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-446676598786 2024-06-03T06:45:23.805 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:23.805 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598786 2024-06-03T06:45:24.951 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 446676598787' 2024-06-03T06:45:24.951 INFO:tasks.workunit.client.0.smithi039.stderr:232671: waiting osd.5 seq 446676598787 2024-06-03T06:45:24.951 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:45:24.951 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598787 -lt 446676598787 2024-06-03T06:45:24.951 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:24.951 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855417 2024-06-03T06:45:24.951 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:24.951 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:45:24.951 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855417 2024-06-03T06:45:24.951 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855417 2024-06-03T06:45:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855417' 2024-06-03T06:45:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:232671: waiting osd.6 seq 150323855417 2024-06-03T06:45:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:45:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855419 -lt 150323855417 2024-06-03T06:45:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691892 2024-06-03T06:45:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:45:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691892 2024-06-03T06:45:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691892 2024-06-03T06:45:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691892' 2024-06-03T06:45:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:232671: waiting osd.7 seq 171798691892 2024-06-03T06:45:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:45:24.953 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691894 -lt 171798691892 2024-06-03T06:45:24.953 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:24.953 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528365 2024-06-03T06:45:24.953 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:24.953 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:45:24.953 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528365 2024-06-03T06:45:24.953 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:24.953 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528365 2024-06-03T06:45:24.953 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528365' 2024-06-03T06:45:24.953 INFO:tasks.workunit.client.0.smithi039.stderr:232671: waiting osd.8 seq 193273528365 2024-06-03T06:45:24.953 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:45:24.953 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528365 -lt 193273528365 2024-06-03T06:45:25.038 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221 echo 'waiting osd.5 seq 446676598786' 2024-06-03T06:45:25.038 INFO:tasks.workunit.client.0.smithi039.stderr:232669: waiting osd.5 seq 446676598786 2024-06-03T06:45:25.038 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:45:25.038 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598787 -lt 446676598786 2024-06-03T06:45:25.038 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:25.038 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855418 2024-06-03T06:45:25.038 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:25.038 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:45:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855418 2024-06-03T06:45:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855418 2024-06-03T06:45:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855418' 2024-06-03T06:45:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:232669: waiting osd.6 seq 150323855418 2024-06-03T06:45:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:45:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855419 -lt 150323855418 2024-06-03T06:45:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691893 2024-06-03T06:45:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:45:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691893 2024-06-03T06:45:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691893 2024-06-03T06:45:25.040 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691893' 2024-06-03T06:45:25.040 INFO:tasks.workunit.client.0.smithi039.stderr:232669: waiting osd.7 seq 171798691893 2024-06-03T06:45:25.040 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:45:25.040 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691894 -lt 171798691893 2024-06-03T06:45:25.040 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:25.040 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528364 2024-06-03T06:45:25.040 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:25.040 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:45:25.040 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528364 2024-06-03T06:45:25.040 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:25.040 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528364 2024-06-03T06:45:25.040 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528364' 2024-06-03T06:45:25.040 INFO:tasks.workunit.client.0.smithi039.stderr:232669: waiting osd.8 seq 193273528364 2024-06-03T06:45:25.040 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:45:25.040 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528365 -lt 193273528364 2024-06-03T06:45:26.359 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:220: flush_pg_stats: for s in $seqs 2024-06-03T06:45:26.359 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631490 2024-06-03T06:45:26.359 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:26.359 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:45:26.359 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631490 2024-06-03T06:45:26.359 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:26.360 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631490 2024-06-03T06:45:26.360 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631490' 2024-06-03T06:45:26.360 INFO:tasks.workunit.client.0.smithi039.stderr:232671: waiting osd.9 seq 442381631490 2024-06-03T06:45:26.360 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:45:26.360 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631492 -lt 442381631490 2024-06-03T06:45:26.360 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:45:26.360 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:26.360 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:26.360 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:45:26.360 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:45:26.360 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:45:26.360 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:45:26.360 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:45:26.360 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:45:26.361 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:45:26.361 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:45:26.361 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:45:26.361 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:45:26.361 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:26.361 INFO:tasks.workunit.client.0.smithi039.stderr:232671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:26.361 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:45:26.361 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:45:26.361 INFO:tasks.workunit.client.0.smithi039.stderr:232671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:45:26.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:45:26.368 INFO:tasks.workunit.client.0.smithi039.stderr:10: flush_pg_stats: for s in $seqs 2024-06-03T06:45:26.369 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631491 2024-06-03T06:45:26.369 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:26.369 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:45:26.369 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631491 2024-06-03T06:45:26.369 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:26.369 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631491 2024-06-03T06:45:26.369 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631491' 2024-06-03T06:45:26.369 INFO:tasks.workunit.client.0.smithi039.stderr:232669: waiting osd.9 seq 442381631491 2024-06-03T06:45:26.369 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:45:26.369 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631492 -lt 442381631491 2024-06-03T06:45:26.370 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:45:26.370 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:26.370 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:26.370 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:45:26.370 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:45:26.370 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:45:26.370 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:45:26.370 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:45:26.370 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:45:26.370 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:45:26.370 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:45:26.370 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:45:26.370 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:45:26.371 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:26.371 INFO:tasks.workunit.client.0.smithi039.stderr:232669: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:26.371 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:45:26.371 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:45:26.371 INFO:tasks.workunit.client.0.smithi039.stderr:232669: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:45:26.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:45:26.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:45:26.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 232668 2024-06-03T06:45:26.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:45:26.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:45:26.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:45:26.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-03T06:45:26.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:45:26.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:45:26.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:45:26.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-06-03T06:45:26.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-03T06:45:26.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-03T06:45:26.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-06-03T06:45:26.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-06-03T06:45:26.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-03T06:45:26.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:45:26.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:45:26.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:45:26.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 235899"' 2024-06-03T06:45:26.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 235899' 2024-06-03T06:45:26.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:45:26.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:45:26.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:45:26.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 235900"' 2024-06-03T06:45:26.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 235900' 2024-06-03T06:45:26.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-03T06:45:26.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:45:26.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 235899 235900' 2024-06-03T06:45:26.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:45:26.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:45:26.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 235899 2024-06-03T06:45:26.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/235902: /' 2024-06-03T06:45:26.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:45:26.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/235904: /' 2024-06-03T06:45:29.307 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:45:29.307 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:45:29.307 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-03T06:45:29.307 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:45:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:45:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:45:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:45:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-03T06:45:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:45:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-03T06:45:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:45:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:45:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:45:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:45:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:45:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:45:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:45:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:45:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:45:29.309 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-03T06:45:29.309 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:45:29.309 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:45:29.309 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-03T06:45:29.309 INFO:tasks.workunit.client.0.smithi039.stderr:235904: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-03T06:45:29.309 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-03T06:45:29.309 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:45:29.309 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:45:29.309 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-03T06:45:29.309 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:45:29.309 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:45:29.309 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:45:29.309 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:45:29.311 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:45:29.311 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:45:29.311 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-03T06:45:29.311 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-03T06:45:29.311 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:45:29.311 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:45:29.312 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:45:29.313 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:45:29.313 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:45:29.313 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-03T06:45:29.313 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-03T06:45:29.313 INFO:tasks.workunit.client.0.smithi039.stderr:235904: start osd.5 2024-06-03T06:45:29.313 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:45:29.383 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:45:29.383 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:45:29.384 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:45:29.384 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:45:29.384 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:45:29.384 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:45:29.384 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:45:29.384 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:45:29.384 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:45:29.384 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:45:29.384 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:45:29.384 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:45:29.384 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:45:29.384 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:45:29.384 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:45:29.385 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:45:29.385 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:45:29.385 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:45:29.385 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:45:29.385 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:45:29.385 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:45:29.385 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:45:29.385 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-03T06:45:29.385 INFO:tasks.workunit.client.0.smithi039.stderr:235902: remove 0#2:eb822e21:::SOMETHING:head# 2024-06-03T06:45:29.385 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:45:29.385 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:45:29.385 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:45:29.385 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:45:29.386 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:45:29.386 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:45:29.386 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:45:29.386 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:45:29.386 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:45:29.386 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:45:29.387 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:45:29.387 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:45:29.387 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:45:29.387 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:45:29.387 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:45:29.387 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:45:29.387 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:45:29.387 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:45:29.387 INFO:tasks.workunit.client.0.smithi039.stderr:235902: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:45:29.387 INFO:tasks.workunit.client.0.smithi039.stderr:235902: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:45:29.387 INFO:tasks.workunit.client.0.smithi039.stderr:235902: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:45:29.387 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:45:29.387 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:45:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:45:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:45:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:45:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:45:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:45:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:45:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:45:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:45:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:45:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:45:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:45:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:235902: start osd.3 2024-06-03T06:45:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:45:34.082 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-03T06:45:34.082 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-03T06:45:34.082 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:45:34.082 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:45:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:45:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 2024-06-03T06:45:29.340+0000 7f754381a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 2024-06-03T06:45:29.343+0000 7f754381a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 2024-06-03T06:45:29.344+0000 7f754381a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-03T06:45:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:45:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-03T06:45:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:45:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:45:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:45:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 0 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 2024-06-03T06:45:29.862+0000 7f754381a640 -1 Falling back to public interface 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 1 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 2 2024-06-03T06:45:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:34.085 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:45:34.085 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 2024-06-03T06:45:32.462+0000 7f754381a640 -1 osd.5 106 log_to_monitors true 2024-06-03T06:45:34.085 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:34.085 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:34.085 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:34.085 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:45:34.085 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 3 2024-06-03T06:45:34.085 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:34.085 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:45:34.313 INFO:tasks.workunit.client.0.smithi039.stderr:235904: osd.5 up in weight 1 up_from 111 up_thru 111 down_at 107 l235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:45:34.313 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:45:34.313 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:45:34.313 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:45:34.314 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:45:34.314 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 2024-06-03T06:45:29.416+0000 7fc78051e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:34.314 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 2024-06-03T06:45:29.419+0000 7fc78051e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:34.314 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 2024-06-03T06:45:29.420+0000 7fc78051e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:34.314 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:45:34.314 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:45:34.314 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:45:34.314 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:45:34.314 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:45:34.314 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:34.314 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:45:34.314 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 0 2024-06-03T06:45:34.315 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:34.315 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:45:34.315 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 2024-06-03T06:45:29.941+0000 7fc78051e640 -1 Falling back to public interface 2024-06-03T06:45:34.315 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:34.315 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:34.315 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:34.315 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:45:34.315 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 1 2024-06-03T06:45:34.315 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:34.315 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:45:34.315 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:34.315 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:34.315 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:34.316 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:45:34.316 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 2 2024-06-03T06:45:34.316 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:34.316 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:45:34.316 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 2024-06-03T06:45:32.570+0000 7fc78051e640 -1 osd.3 106 log_to_monitors true 2024-06-03T06:45:34.316 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:34.316 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:34.316 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:34.316 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:45:34.316 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 3 2024-06-03T06:45:34.316 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:34.316 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:45:35.003 INFO:tasks.workunit.client.0.smithi039.stderr:235902: osd.3 up in weight 1 up_from 111 up_thru 111 down_at 107 last_clean_interval [104,106) [v2:127.0.0.1:6826/1414772049,v1:127.0.0.1:6827/1414772049] [v2:127.0.0.1:6828/1414772049,v1:127.0.0.1:6829/1414772049] exists,up c849e020-3670-4066-b986-5db1f468a8d7 2024-06-03T06:45:35.003 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:45:35.003 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:45:35.003 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:45:35.003 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:45:35.003 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:45:35.003 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:45:35.003 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:45:35.003 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:45:35.004 INFO:tasks.workunit.client.0.smithi039.stderr:235904: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:45:35.004 INFO:tasks.workunit.client.0.smithi039.stderr:235904: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:45:35.004 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:45:35.004 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:45:35.004 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:45:35.004 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:45:35.004 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:45:35.004 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:45:35.004 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:45:35.004 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:45:35.004 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:45:35.004 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:45:35.004 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 1 2024-06-03T06:45:35.004 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 2 2024-06-03T06:45:35.005 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 3 2024-06-03T06:45:35.005 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 4 2024-06-03T06:45:35.005 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 5 2024-06-03T06:45:35.005 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 6 2024-06-03T06:45:35.005 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 7 2024-06-03T06:45:35.005 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 8 2024-06-03T06:45:35.005 INFO:tasks.workunit.client.0.smithi039.stderr:235904: 9' 2024-06-03T06:45:35.005 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:45:35.005 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:35.005 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:45:35.005 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836580 2024-06-03T06:45:35.005 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836580 2024-06-03T06:45:35.006 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580' 2024-06-03T06:45:35.006 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:35.006 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:45:35.006 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673055 2024-06-03T06:45:35.006 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673055 2024-06-03T06:45:35.006 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673055' 2024-06-03T06:45:35.006 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:35.193 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.clieast_clean_interval [72,106) [v2:127.0.0.1:6874/998923363,v1:127.0.0.1:6875/998923363] [v2:127.0.0.1:6876/998923363,v1:127.0.0.1:6877/998923363] exists,up a251791b-8f01-466a-8f68-ddc09d9720c7 2024-06-03T06:45:35.193 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:45:35.193 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:45:35.193 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:45:35.193 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:45:35.193 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:45:35.193 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:45:35.193 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:45:35.194 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:45:35.194 INFO:tasks.workunit.client.0.smithi039.stderr:235902: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:45:35.194 INFO:tasks.workunit.client.0.smithi039.stderr:235902: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:45:35.194 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:45:35.194 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:45:35.194 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:45:35.194 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:45:35.194 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:45:35.194 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:45:35.194 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:45:35.194 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:45:35.194 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:45:35.195 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:45:35.195 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 1 2024-06-03T06:45:35.195 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 2 2024-06-03T06:45:35.195 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 3 2024-06-03T06:45:35.195 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 4 2024-06-03T06:45:35.195 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 5 2024-06-03T06:45:35.195 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 6 2024-06-03T06:45:35.195 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 7 2024-06-03T06:45:35.195 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 8 2024-06-03T06:45:35.195 INFO:tasks.workunit.client.0.smithi039.stderr:235902: 9' 2024-06-03T06:45:35.195 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:45:35.195 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:35.196 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:45:35.196 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836581 2024-06-03T06:45:35.196 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836581 2024-06-03T06:45:35.196 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581' 2024-06-03T06:45:35.196 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:35.196 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:45:35.196 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673056 2024-06-03T06:45:35.196 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673056 2024-06-03T06:45:35.196 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673056' 2024-06-03T06:45:35.196 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:35.931 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/nt.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:45:35.931 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509529 2024-06-03T06:45:35.931 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509529 2024-06-03T06:45:35.931 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673055 2-64424509529' 2024-06-03T06:45:35.931 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:35.931 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:45:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369858 2024-06-03T06:45:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369858 2024-06-03T06:45:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673055 2-64424509529 3-476741369858' 2024-06-03T06:45:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:45:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182475 2024-06-03T06:45:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182475 2024-06-03T06:45:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673055 2-64424509529 3-476741369858 4-107374182475' 2024-06-03T06:45:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:45:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369858 2024-06-03T06:45:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369858 2024-06-03T06:45:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673055 2-64424509529 3-476741369858 4-107374182475 5-476741369858' 2024-06-03T06:45:35.933 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:35.933 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:45:35.933 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855422 2024-06-03T06:45:35.933 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855422 2024-06-03T06:45:35.933 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673055 2-64424509529 3-476741369858 4-107374182475 5-476741369858 6-150323855422' 2024-06-03T06:45:35.933 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:35.933 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:45:35.933 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691897 2024-06-03T06:45:35.933 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691897 2024-06-03T06:45:35.933 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673055 2-64424509529 3-476741369858 4-107374182475 5-476741369858 6-150323855422 7-171798691897' 2024-06-03T06:45:35.933 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:35.933 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:45:36.042 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtesqa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:45:36.042 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509530 2024-06-03T06:45:36.042 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509530 2024-06-03T06:45:36.042 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673056 2-64424509530' 2024-06-03T06:45:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:45:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369859 2024-06-03T06:45:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369859 2024-06-03T06:45:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673056 2-64424509530 3-476741369859' 2024-06-03T06:45:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:45:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182476 2024-06-03T06:45:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182476 2024-06-03T06:45:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673056 2-64424509530 3-476741369859 4-107374182476' 2024-06-03T06:45:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:45:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369859 2024-06-03T06:45:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369859 2024-06-03T06:45:36.044 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673056 2-64424509530 3-476741369859 4-107374182476 5-476741369859' 2024-06-03T06:45:36.044 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:36.044 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:45:36.044 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855423 2024-06-03T06:45:36.044 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855423 2024-06-03T06:45:36.044 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673056 2-64424509530 3-476741369859 4-107374182476 5-476741369859 6-150323855423' 2024-06-03T06:45:36.044 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:36.044 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:45:36.044 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691898 2024-06-03T06:45:36.044 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691898 2024-06-03T06:45:36.044 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673056 2-64424509530 3-476741369859 4-107374182476 5-476741369859 6-150323855423 7-171798691898' 2024-06-03T06:45:36.044 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:36.044 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:45:36.807 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clot/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528369 2024-06-03T06:45:36.807 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528369 2024-06-03T06:45:36.807 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673055 2-64424509529 3-476741369858 4-107374182475 5-476741369858 6-150323855422 7-171798691897 8-193273528369' 2024-06-03T06:45:36.807 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:45:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631495 2024-06-03T06:45:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631495 2024-06-03T06:45:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673055 2-64424509529 3-476741369858 4-107374182475 5-476741369858 6-150323855422 7-171798691897 8-193273528369 9-442381631495' 2024-06-03T06:45:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836580 2024-06-03T06:45:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:45:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836580 2024-06-03T06:45:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836580 2024-06-03T06:45:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-06-03T06:45:36.809 INFO:tasks.workunit.client.0.smithi039.stderr:235904: waiting osd.0 seq 21474836580 2024-06-03T06:45:36.809 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:45:36.809 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836581 -lt 21474836580 2024-06-03T06:45:36.809 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:36.809 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673055 2024-06-03T06:45:36.809 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:36.809 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:45:36.809 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673055 2024-06-03T06:45:36.809 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:36.809 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673055 2024-06-03T06:45:36.809 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-06-03T06:45:36.809 INFO:tasks.workunit.client.0.smithi039.stderr:235904: waiting osd.1 seq 42949673055 2024-06-03T06:45:36.810 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:45:36.810 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673057 -lt 42949673055 2024-06-03T06:45:36.810 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:36.810 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509529 2024-06-03T06:45:36.810 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:36.810 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:45:36.880 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/ne.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528370 2024-06-03T06:45:36.880 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528370 2024-06-03T06:45:36.881 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673056 2-64424509530 3-476741369859 4-107374182476 5-476741369859 6-150323855423 7-171798691898 8-193273528370' 2024-06-03T06:45:36.881 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:36.881 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:45:36.881 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631496 2024-06-03T06:45:36.881 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631496 2024-06-03T06:45:36.881 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673056 2-64424509530 3-476741369859 4-107374182476 5-476741369859 6-150323855423 7-171798691898 8-193273528370 9-442381631496' 2024-06-03T06:45:36.881 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:36.881 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836581 2024-06-03T06:45:36.881 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:36.881 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:45:36.881 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836581 2024-06-03T06:45:36.881 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:36.882 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836581 2024-06-03T06:45:36.882 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-06-03T06:45:36.882 INFO:tasks.workunit.client.0.smithi039.stderr:235902: waiting osd.0 seq 21474836581 2024-06-03T06:45:36.882 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:45:36.882 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836581 -lt 21474836581 2024-06-03T06:45:36.882 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:36.882 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673056 2024-06-03T06:45:36.882 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:36.882 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:45:36.882 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673056 2024-06-03T06:45:36.882 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:36.882 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673056 2024-06-03T06:45:36.882 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-06-03T06:45:36.883 INFO:tasks.workunit.client.0.smithi039.stderr:235902: waiting osd.1 seq 42949673056 2024-06-03T06:45:36.883 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:45:36.883 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673057 -lt 42949673056 2024-06-03T06:45:36.883 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:36.883 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509530 2024-06-03T06:45:36.883 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:36.883 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:45:37.756 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/cloneclone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509529 2024-06-03T06:45:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509529 2024-06-03T06:45:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-06-03T06:45:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:235904: waiting osd.2 seq 64424509529 2024-06-03T06:45:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:45:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509530 -lt 64424509529 2024-06-03T06:45:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-476741369858 2024-06-03T06:45:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:45:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-476741369858 2024-06-03T06:45:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369858 2024-06-03T06:45:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 476741369858' 2024-06-03T06:45:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:235904: waiting osd.3 seq 476741369858 2024-06-03T06:45:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:45:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369859 -lt 476741369858 2024-06-03T06:45:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182475 2024-06-03T06:45:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:45:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182475 2024-06-03T06:45:37.759 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:37.759 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182475 2024-06-03T06:45:37.759 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182475' 2024-06-03T06:45:37.759 INFO:tasks.workunit.client.0.smithi039.stderr:235904: waiting osd.4 seq 107374182475 2024-06-03T06:45:37.759 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:45:37.759 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182476 -lt 107374182475 2024-06-03T06:45:37.759 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:37.759 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-476741369858 2024-06-03T06:45:37.759 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:37.759 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:45:37.759 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-476741369858 2024-06-03T06:45:37.759 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:37.759 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369858 2024-06-03T06:45:37.793 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509530 2024-06-03T06:45:37.793 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:37.793 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509530 2024-06-03T06:45:37.793 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-06-03T06:45:37.793 INFO:tasks.workunit.client.0.smithi039.stderr:235902: waiting osd.2 seq 64424509530 2024-06-03T06:45:37.794 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:45:37.794 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509530 -lt 64424509530 2024-06-03T06:45:37.794 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:37.794 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-476741369859 2024-06-03T06:45:37.794 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:37.794 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:45:37.794 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-476741369859 2024-06-03T06:45:37.794 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:37.794 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369859 2024-06-03T06:45:37.794 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 476741369859' 2024-06-03T06:45:37.794 INFO:tasks.workunit.client.0.smithi039.stderr:235902: waiting osd.3 seq 476741369859 2024-06-03T06:45:37.794 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:45:37.795 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369859 -lt 476741369859 2024-06-03T06:45:37.795 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:37.795 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182476 2024-06-03T06:45:37.795 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:37.795 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:45:37.795 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182476 2024-06-03T06:45:37.795 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:37.795 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182476 2024-06-03T06:45:37.795 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182476' 2024-06-03T06:45:37.795 INFO:tasks.workunit.client.0.smithi039.stderr:235902: waiting osd.4 seq 107374182476 2024-06-03T06:45:37.795 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:45:37.795 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182476 -lt 107374182476 2024-06-03T06:45:37.795 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:37.796 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-476741369859 2024-06-03T06:45:37.796 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:37.796 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:45:37.796 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-476741369859 2024-06-03T06:45:37.796 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:37.796 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369859 2024-06-03T06:45:38.998 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'wai 'waiting osd.5 seq 476741369858' 2024-06-03T06:45:38.999 INFO:tasks.workunit.client.0.smithi039.stderr:235904: waiting osd.5 seq 476741369858 2024-06-03T06:45:38.999 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:45:38.999 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369859 -lt 476741369858 2024-06-03T06:45:38.999 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:38.999 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855422 2024-06-03T06:45:38.999 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:38.999 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:45:38.999 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855422 2024-06-03T06:45:38.999 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:38.999 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855422 2024-06-03T06:45:38.999 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855422' 2024-06-03T06:45:38.999 INFO:tasks.workunit.client.0.smithi039.stderr:235904: waiting osd.6 seq 150323855422 2024-06-03T06:45:38.999 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:45:38.999 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855424 -lt 150323855422 2024-06-03T06:45:39.000 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:39.000 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691897 2024-06-03T06:45:39.000 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:39.000 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:45:39.000 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691897 2024-06-03T06:45:39.000 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:39.000 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691897 2024-06-03T06:45:39.000 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691897' 2024-06-03T06:45:39.000 INFO:tasks.workunit.client.0.smithi039.stderr:235904: waiting osd.7 seq 171798691897 2024-06-03T06:45:39.000 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:45:39.000 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691899 -lt 171798691897 2024-06-03T06:45:39.000 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:39.001 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528369 2024-06-03T06:45:39.001 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:39.001 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:45:39.001 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528369 2024-06-03T06:45:39.001 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:39.001 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528369 2024-06-03T06:45:39.001 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-06-03T06:45:39.001 INFO:tasks.workunit.client.0.smithi039.stderr:235904: waiting osd.8 seq 193273528369 2024-06-03T06:45:39.001 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:45:39.001 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528370 -lt 193273528369 2024-06-03T06:45:39.070 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fting osd.5 seq 476741369859' 2024-06-03T06:45:39.070 INFO:tasks.workunit.client.0.smithi039.stderr:235902: waiting osd.5 seq 476741369859 2024-06-03T06:45:39.070 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:45:39.070 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369859 -lt 476741369859 2024-06-03T06:45:39.070 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:39.070 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855423 2024-06-03T06:45:39.070 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:39.070 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:45:39.070 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855423 2024-06-03T06:45:39.070 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:39.070 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855423 2024-06-03T06:45:39.070 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855423' 2024-06-03T06:45:39.071 INFO:tasks.workunit.client.0.smithi039.stderr:235902: waiting osd.6 seq 150323855423 2024-06-03T06:45:39.071 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:45:39.071 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855424 -lt 150323855423 2024-06-03T06:45:39.071 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:39.071 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691898 2024-06-03T06:45:39.071 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:39.071 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:45:39.071 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691898 2024-06-03T06:45:39.071 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:39.071 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691898 2024-06-03T06:45:39.071 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691898' 2024-06-03T06:45:39.071 INFO:tasks.workunit.client.0.smithi039.stderr:235902: waiting osd.7 seq 171798691898 2024-06-03T06:45:39.071 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:45:39.072 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691899 -lt 171798691898 2024-06-03T06:45:39.072 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:39.072 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528370 2024-06-03T06:45:39.072 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:39.072 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:45:39.072 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528370 2024-06-03T06:45:39.072 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:39.072 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528370 2024-06-03T06:45:39.072 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528370' 2024-06-03T06:45:39.072 INFO:tasks.workunit.client.0.smithi039.stderr:235902: waiting osd.8 seq 193273528370 2024-06-03T06:45:39.072 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:45:39.072 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528370 -lt 193273528370 2024-06-03T06:45:40.383 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_lush_pg_stats: for s in $seqs 2024-06-03T06:45:40.383 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631495 2024-06-03T06:45:40.383 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:40.383 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:45:40.383 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631495 2024-06-03T06:45:40.383 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:40.383 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631495 2024-06-03T06:45:40.383 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631495' 2024-06-03T06:45:40.383 INFO:tasks.workunit.client.0.smithi039.stderr:235904: waiting osd.9 seq 442381631495 2024-06-03T06:45:40.383 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:45:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631496 -lt 442381631495 2024-06-03T06:45:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:45:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:45:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:45:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:45:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:45:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:45:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:45:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:45:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:45:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:45:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:45:40.385 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:40.385 INFO:tasks.workunit.client.0.smithi039.stderr:235904: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:40.385 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:45:40.385 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:45:40.385 INFO:tasks.workunit.client.0.smithi039.stderr:235904: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:45:40.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:45:40.466 INFO:tasks.workunit.client.0.smithi039.stderr:pg_stats: for s in $seqs 2024-06-03T06:45:40.466 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631496 2024-06-03T06:45:40.466 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:40.466 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:45:40.467 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631496 2024-06-03T06:45:40.467 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:40.467 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631496 2024-06-03T06:45:40.467 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631496' 2024-06-03T06:45:40.467 INFO:tasks.workunit.client.0.smithi039.stderr:235902: waiting osd.9 seq 442381631496 2024-06-03T06:45:40.467 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:45:40.467 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631496 -lt 442381631496 2024-06-03T06:45:40.467 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:45:40.467 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:40.467 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:40.467 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:45:40.467 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:45:40.468 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:45:40.468 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:45:40.468 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:45:40.468 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:45:40.468 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:45:40.468 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:45:40.468 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:45:40.468 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:45:40.468 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:40.468 INFO:tasks.workunit.client.0.smithi039.stderr:235902: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:40.468 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:45:40.468 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:45:40.468 INFO:tasks.workunit.client.0.smithi039.stderr:235902: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:45:40.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:45:40.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:45:40.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 235900 2024-06-03T06:45:40.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:45:40.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:45:40.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:45:40.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-03T06:45:40.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:45:40.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-03T06:45:40.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:45:40.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:45:40.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:45:40.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:45:40.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-03T06:45:40.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-03T06:45:40.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:45:40.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:45:40.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:45:40.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:45:40.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:45:40.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:45:41.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:45:07.692090+0000 2024-06-03T06:45:41.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:45:41.371 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:45:41.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:45:07.692090+0000 2024-06-03T06:45:41.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:45:41.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:45:07.692090+0000 2024-06-03T06:45:41.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:45:41.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:45:41.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:45:41.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:45:41.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:45:41.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:45:41.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:45:41.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:45:41.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:45:07.692090+0000 '>' 2024-06-03T06:45:07.692090+0000 2024-06-03T06:45:41.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:45:42.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:45:42.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:45:42.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:45:42.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:45:42.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:45:42.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:45:42.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:45:42.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:45:07.692090+0000 '>' 2024-06-03T06:45:07.692090+0000 2024-06-03T06:45:42.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:45:43.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:45:43.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:45:43.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:45:43.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:45:43.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:45:43.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:45:43.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:45:44.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:45:41.579183+0000 '>' 2024-06-03T06:45:07.692090+0000 2024-06-03T06:45:44.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:45:44.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-03T06:45:44.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:45:44.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:45:44.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:45:44.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 239255"' 2024-06-03T06:45:44.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 239255' 2024-06-03T06:45:44.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:45:44.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:45:44.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:45:44.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 239257"' 2024-06-03T06:45:44.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:45:44.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 239257' 2024-06-03T06:45:44.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-03T06:45:44.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 239255 239257' 2024-06-03T06:45:44.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:45:44.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:45:44.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 239255 2024-06-03T06:45:44.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/239258: /' 2024-06-03T06:45:44.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:45:44.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/239260: /' 2024-06-03T06:45:47.151 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:45:47.152 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:45:47.152 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:45:47.152 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:45:47.152 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:45:47.152 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:45:47.152 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:45:47.152 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:45:47.152 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:45:47.152 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:45:47.152 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:45:47.152 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:45:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:45:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:45:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:45:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:45:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:45:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:45:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:45:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:45:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:45:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:45:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-03T06:45:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:239258: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-03T06:45:47.153 INFO:tasks.workunit.client.0.smithi039.stderr:239258: _ 2024-06-03T06:45:47.154 INFO:tasks.workunit.client.0.smithi039.stderr:239258: hinfo_key 2024-06-03T06:45:47.154 INFO:tasks.workunit.client.0.smithi039.stderr:239258: snapset 2024-06-03T06:45:47.154 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:45:47.154 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:45:47.154 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:45:47.154 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:45:47.154 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:45:47.154 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:45:47.154 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:45:47.155 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:45:47.155 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:45:47.155 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:45:47.155 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:45:47.155 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:45:47.155 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:45:47.155 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:45:47.156 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:45:47.156 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:45:47.156 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:45:47.156 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:45:47.156 INFO:tasks.workunit.client.0.smithi039.stderr:239258: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:45:47.156 INFO:tasks.workunit.client.0.smithi039.stderr:239258: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:45:47.156 INFO:tasks.workunit.client.0.smithi039.stderr:239258: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:45:47.156 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:45:47.156 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:45:47.156 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:45:47.156 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:45:47.156 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:45:47.156 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:45:47.156 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:45:47.157 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:45:47.157 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:45:47.157 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:45:47.157 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:45:47.157 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:45:47.157 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:45:47.157 INFO:tasks.workunit.client.0.smithi039.stderr:239258: start osd.3 2024-06-03T06:45:47.638 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:45:47.638 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:45:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-03T06:45:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:45:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:45:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:45:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:45:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-03T06:45:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:45:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-03T06:45:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:45:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:45:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:45:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:45:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:45:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:45:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:45:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:45:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:45:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-03T06:45:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:45:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:45:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-06-03T06:45:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:239260: _ 2024-06-03T06:45:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:239260: hinfo_key 2024-06-03T06:45:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:239260: snapset 2024-06-03T06:45:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-03T06:45:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:45:47.641 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:45:47.641 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-03T06:45:47.641 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:45:47.641 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:45:47.641 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:45:47.641 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:45:47.641 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:45:47.641 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:45:47.641 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-03T06:45:47.642 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-03T06:45:47.642 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:45:47.642 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:45:47.642 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:45:47.642 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:45:47.642 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:45:47.642 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:45:47.642 INFO:tasks.workunit.client.0.smithi039.stderr:239260: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:45:47.642 INFO:tasks.workunit.client.0.smithi039.stderr:239260: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:45:47.642 INFO:tasks.workunit.client.0.smithi039.stderr:239260: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:45:47.642 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:45:47.642 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:45:47.643 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:45:47.643 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:45:47.643 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:45:47.643 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:45:47.643 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:45:47.643 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:45:47.643 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:45:47.643 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:45:47.643 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:45:47.643 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-03T06:45:47.643 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-03T06:45:47.644 INFO:tasks.workunit.client.0.smithi039.stderr:239260: start osd.5 2024-06-03T06:45:51.342 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:45:51.342 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:45:51.342 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:45:51.342 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:45:51.343 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:45:51.343 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:45:51.343 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 2024-06-03T06:45:47.184+0000 7f8556d25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:51.343 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 2024-06-03T06:45:47.215+0000 7f8556d25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:51.343 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 2024-06-03T06:45:47.216+0000 7f8556d25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:51.343 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:45:51.343 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:45:51.343 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:45:51.343 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:45:51.343 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:45:51.343 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:51.343 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:45:51.343 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 0 2024-06-03T06:45:51.343 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 2024-06-03T06:45:47.738+0000 7f8556d25640 -1 Falling back to public interface 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 1 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 2 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:51.344 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:45:51.345 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 2024-06-03T06:45:50.122+0000 7f8556d25640 -1 osd.3 112 log_to_monitors true 2024-06-03T06:45:51.345 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:51.345 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:51.345 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:51.345 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:45:51.345 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 3 2024-06-03T06:45:51.345 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:52.205 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-06-03T06:45:52.205 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-03T06:45:52.205 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-03T06:45:52.205 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:45:52.205 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:45:52.205 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:45:52.206 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 2024-06-03T06:45:47.667+0000 7fb812430640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:52.206 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 2024-06-03T06:45:47.670+0000 7fb812430640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:52.206 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 2024-06-03T06:45:47.671+0000 7fb812430640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:45:52.206 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-03T06:45:52.206 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:45:52.206 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-03T06:45:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:45:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:45:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:45:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 0 2024-06-03T06:45:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:45:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 2024-06-03T06:45:48.193+0000 7fb812430640 -1 Falling back to public interface 2024-06-03T06:45:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:45:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 1 2024-06-03T06:45:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:45:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:45:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 2 2024-06-03T06:45:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:45:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:45:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 2024-06-03T06:45:50.922+0000 7fb812430640 -1 osd.5 112 log_to_monitors true 2024-06-03T06:45:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:45:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:45:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:45:52.209 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 3 2024-06-03T06:45:52.209 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:45:52.209 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:45:52.636 INFO:tasks.workunit.client.0.smithi039.stderr:239260: osd.5 up in weight 1 up_from 119 up_thru 111 dowrs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:45:52.636 INFO:tasks.workunit.client.0.smithi039.stderr:239258: osd.3 up in weight 1 up_from 116 up_thru 111 down_at 113 last_clean_interval [111,112) [v2:127.0.0.1:6826/590442640,v1:127.0.0.1:6827/590442640] [v2:127.0.0.1:6828/590442640,v1:127.0.0.1:6829/590442640] exists,up a251791b-8f01-466a-8f68-ddc09d9720c7 2024-06-03T06:45:52.636 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:45:52.636 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:45:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:45:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:45:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:45:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:45:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:45:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:45:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:239258: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:45:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:239258: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:45:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:45:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:45:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:45:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:45:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:45:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:45:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:45:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:45:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:45:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:45:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 1 2024-06-03T06:45:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 2 2024-06-03T06:45:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 3 2024-06-03T06:45:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 4 2024-06-03T06:45:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 5 2024-06-03T06:45:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 6 2024-06-03T06:45:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 7 2024-06-03T06:45:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 8 2024-06-03T06:45:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:239258: 9' 2024-06-03T06:45:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:45:52.639 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:52.639 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:45:52.639 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836585 2024-06-03T06:45:52.639 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836585 2024-06-03T06:45:52.639 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585' 2024-06-03T06:45:52.639 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:52.639 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:45:52.639 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673061 2024-06-03T06:45:52.639 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673061 2024-06-03T06:45:52.639 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673061' 2024-06-03T06:45:53.441 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/n_at 113 last_clean_interval [111,112) [v2:127.0.0.1:6874/3847944282,v1:127.0.0.1:6875/3847944282] [v2:127.0.0.1:6876/3847944282,v1:127.0.0.1:6877/3847944282] exists,up c849e020-3670-4066-b986-5db1f468a8d7 2024-06-03T06:45:53.441 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:45:53.441 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:45:53.441 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:45:53.441 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:45:53.441 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:45:53.441 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:45:53.441 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:45:53.442 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:45:53.442 INFO:tasks.workunit.client.0.smithi039.stderr:239260: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:45:53.442 INFO:tasks.workunit.client.0.smithi039.stderr:239260: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:45:53.442 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:45:53.442 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:45:53.442 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:45:53.442 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:45:53.442 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:45:53.442 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:45:53.442 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:45:53.442 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:45:53.442 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:45:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:45:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 1 2024-06-03T06:45:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 2 2024-06-03T06:45:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 3 2024-06-03T06:45:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 4 2024-06-03T06:45:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 5 2024-06-03T06:45:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 6 2024-06-03T06:45:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 7 2024-06-03T06:45:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 8 2024-06-03T06:45:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:239260: 9' 2024-06-03T06:45:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:45:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:45:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836586 2024-06-03T06:45:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836586 2024-06-03T06:45:53.444 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586' 2024-06-03T06:45:53.444 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:53.444 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:45:53.444 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673062 2024-06-03T06:45:53.444 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673062 2024-06-03T06:45:53.444 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673062' 2024-06-03T06:45:53.444 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:53.978 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:53.978 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:45:53.978 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509535 2024-06-03T06:45:53.978 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509535 2024-06-03T06:45:53.978 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509535' 2024-06-03T06:45:53.978 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:53.978 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:45:53.978 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-06-03T06:45:53.978 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-06-03T06:45:53.978 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509535 3-498216206338' 2024-06-03T06:45:53.979 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:53.979 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:45:53.979 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182481 2024-06-03T06:45:53.979 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182481 2024-06-03T06:45:53.979 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509535 3-498216206338 4-107374182481' 2024-06-03T06:45:53.979 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:53.979 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:45:53.979 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=511101108226 2024-06-03T06:45:53.979 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 511101108226 2024-06-03T06:45:53.979 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509535 3-498216206338 4-107374182481 5-511101108226' 2024-06-03T06:45:53.979 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:53.979 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:45:53.979 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855428 2024-06-03T06:45:53.979 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855428 2024-06-03T06:45:53.980 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509535 3-498216206338 4-107374182481 5-511101108226 6-150323855428' 2024-06-03T06:45:53.980 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:53.980 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:45:53.980 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691903 2024-06-03T06:45:53.980 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691903 2024-06-03T06:45:53.980 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509535 3-498216206338 4-107374182481 5-511101108226 6-150323855428 7-171798691903' 2024-06-03T06:45:53.980 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:54.648 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/stclone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:45:54.648 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509536 2024-06-03T06:45:54.648 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509536 2024-06-03T06:45:54.648 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509536' 2024-06-03T06:45:54.649 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:54.649 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:45:54.649 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206339 2024-06-03T06:45:54.649 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206339 2024-06-03T06:45:54.649 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509536 3-498216206339' 2024-06-03T06:45:54.649 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:54.649 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:45:54.649 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182482 2024-06-03T06:45:54.649 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182482 2024-06-03T06:45:54.649 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509536 3-498216206339 4-107374182482' 2024-06-03T06:45:54.649 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:54.649 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:45:54.649 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=511101108227 2024-06-03T06:45:54.649 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 511101108227 2024-06-03T06:45:54.650 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509536 3-498216206339 4-107374182482 5-511101108227' 2024-06-03T06:45:54.650 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:54.650 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:45:54.650 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855429 2024-06-03T06:45:54.650 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855429 2024-06-03T06:45:54.650 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509536 3-498216206339 4-107374182482 5-511101108227 6-150323855429' 2024-06-03T06:45:54.650 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:54.650 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:45:54.650 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691904 2024-06-03T06:45:54.650 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691904 2024-06-03T06:45:54.650 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509536 3-498216206339 4-107374182482 5-511101108227 6-150323855429 7-171798691904' 2024-06-03T06:45:54.650 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:54.650 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:45:55.023 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubunandalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:45:55.023 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528374 2024-06-03T06:45:55.023 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528374 2024-06-03T06:45:55.023 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509535 3-498216206338 4-107374182481 5-511101108226 6-150323855428 7-171798691903 8-193273528374' 2024-06-03T06:45:55.023 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:55.023 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:45:55.024 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631501 2024-06-03T06:45:55.024 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631501 2024-06-03T06:45:55.024 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509535 3-498216206338 4-107374182481 5-511101108226 6-150323855428 7-171798691903 8-193273528374 9-442381631501' 2024-06-03T06:45:55.024 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:55.024 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836585 2024-06-03T06:45:55.024 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:55.024 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:45:55.024 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836585 2024-06-03T06:45:55.024 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:55.024 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836585 2024-06-03T06:45:55.024 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-06-03T06:45:55.024 INFO:tasks.workunit.client.0.smithi039.stderr:239258: waiting osd.0 seq 21474836585 2024-06-03T06:45:55.024 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:45:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836587 -lt 21474836585 2024-06-03T06:45:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673061 2024-06-03T06:45:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:45:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673061 2024-06-03T06:45:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673061 2024-06-03T06:45:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-06-03T06:45:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:239258: waiting osd.1 seq 42949673061 2024-06-03T06:45:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:45:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673062 -lt 42949673061 2024-06-03T06:45:55.025 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:55.026 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509535 2024-06-03T06:45:55.026 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:55.386 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtestu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528375 2024-06-03T06:45:55.386 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528375 2024-06-03T06:45:55.386 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509536 3-498216206339 4-107374182482 5-511101108227 6-150323855429 7-171798691904 8-193273528375' 2024-06-03T06:45:55.386 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:45:55.386 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:45:55.386 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631502 2024-06-03T06:45:55.386 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631502 2024-06-03T06:45:55.386 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509536 3-498216206339 4-107374182482 5-511101108227 6-150323855429 7-171798691904 8-193273528375 9-442381631502' 2024-06-03T06:45:55.387 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:55.387 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836586 2024-06-03T06:45:55.387 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:55.387 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:45:55.387 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836586 2024-06-03T06:45:55.387 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:55.387 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836586 2024-06-03T06:45:55.387 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2024-06-03T06:45:55.387 INFO:tasks.workunit.client.0.smithi039.stderr:239260: waiting osd.0 seq 21474836586 2024-06-03T06:45:55.388 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:45:55.388 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836587 -lt 21474836586 2024-06-03T06:45:55.388 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:55.388 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673062 2024-06-03T06:45:55.388 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:55.388 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:45:55.388 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673062 2024-06-03T06:45:55.388 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:55.388 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673062 2024-06-03T06:45:55.388 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-06-03T06:45:55.389 INFO:tasks.workunit.client.0.smithi039.stderr:239260: waiting osd.1 seq 42949673062 2024-06-03T06:45:55.389 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:45:55.389 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673062 -lt 42949673062 2024-06-03T06:45:55.389 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:55.389 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509536 2024-06-03T06:45:55.389 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:55.389 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:45:55.931 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntut/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:45:55.931 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509535 2024-06-03T06:45:55.931 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:55.931 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509535 2024-06-03T06:45:55.932 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509535' 2024-06-03T06:45:55.932 INFO:tasks.workunit.client.0.smithi039.stderr:239258: waiting osd.2 seq 64424509535 2024-06-03T06:45:55.932 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:45:55.932 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509536 -lt 64424509535 2024-06-03T06:45:55.932 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:55.932 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-498216206338 2024-06-03T06:45:55.932 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:55.932 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:45:55.932 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-498216206338 2024-06-03T06:45:55.932 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:55.932 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-06-03T06:45:55.932 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 498216206338' 2024-06-03T06:45:55.932 INFO:tasks.workunit.client.0.smithi039.stderr:239258: waiting osd.3 seq 498216206338 2024-06-03T06:45:55.932 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:45:55.933 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206339 -lt 498216206338 2024-06-03T06:45:55.933 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:55.933 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182481 2024-06-03T06:45:55.933 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:55.933 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:45:55.933 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182481 2024-06-03T06:45:55.933 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:55.933 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182481 2024-06-03T06:45:55.933 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-06-03T06:45:55.933 INFO:tasks.workunit.client.0.smithi039.stderr:239258: waiting osd.4 seq 107374182481 2024-06-03T06:45:55.933 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:45:55.933 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182482 -lt 107374182481 2024-06-03T06:45:55.933 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:55.933 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-511101108226 2024-06-03T06:45:55.934 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:55.934 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:45:55.934 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-511101108226 2024-06-03T06:45:55.934 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:56.314 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51110110/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509536 2024-06-03T06:45:56.314 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:56.314 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509536 2024-06-03T06:45:56.314 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-06-03T06:45:56.314 INFO:tasks.workunit.client.0.smithi039.stderr:239260: waiting osd.2 seq 64424509536 2024-06-03T06:45:56.314 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:45:56.314 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509536 -lt 64424509536 2024-06-03T06:45:56.314 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:56.314 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-498216206339 2024-06-03T06:45:56.314 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:56.314 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:45:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-498216206339 2024-06-03T06:45:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206339 2024-06-03T06:45:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 498216206339' 2024-06-03T06:45:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:239260: waiting osd.3 seq 498216206339 2024-06-03T06:45:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:45:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206339 -lt 498216206339 2024-06-03T06:45:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182482 2024-06-03T06:45:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:45:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182482 2024-06-03T06:45:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182482 2024-06-03T06:45:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182482' 2024-06-03T06:45:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:239260: waiting osd.4 seq 107374182482 2024-06-03T06:45:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:45:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182482 -lt 107374182482 2024-06-03T06:45:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-511101108227 2024-06-03T06:45:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:45:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-511101108227 2024-06-03T06:45:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=511101108227 2024-06-03T06:45:57.170 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_st8226 2024-06-03T06:45:57.170 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 511101108226' 2024-06-03T06:45:57.170 INFO:tasks.workunit.client.0.smithi039.stderr:239258: waiting osd.5 seq 511101108226 2024-06-03T06:45:57.170 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:45:57.170 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 511101108227 -lt 511101108226 2024-06-03T06:45:57.170 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:57.170 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855428 2024-06-03T06:45:57.170 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:45:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855428 2024-06-03T06:45:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855428 2024-06-03T06:45:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855428' 2024-06-03T06:45:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:239258: waiting osd.6 seq 150323855428 2024-06-03T06:45:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:45:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855429 -lt 150323855428 2024-06-03T06:45:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691903 2024-06-03T06:45:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:45:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691903 2024-06-03T06:45:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691903 2024-06-03T06:45:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691903' 2024-06-03T06:45:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:239258: waiting osd.7 seq 171798691903 2024-06-03T06:45:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:45:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691904 -lt 171798691903 2024-06-03T06:45:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528374 2024-06-03T06:45:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:45:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528374 2024-06-03T06:45:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528374 2024-06-03T06:45:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528374' 2024-06-03T06:45:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:239258: waiting osd.8 seq 193273528374 2024-06-03T06:45:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:45:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 19327ats: echo 'waiting osd.5 seq 511101108227' 2024-06-03T06:45:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:239260: waiting osd.5 seq 511101108227 2024-06-03T06:45:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:45:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 511101108227 -lt 511101108227 2024-06-03T06:45:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855429 2024-06-03T06:45:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:45:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855429 2024-06-03T06:45:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855429 2024-06-03T06:45:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855429' 2024-06-03T06:45:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:239260: waiting osd.6 seq 150323855429 2024-06-03T06:45:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:45:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855429 -lt 150323855429 2024-06-03T06:45:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691904 2024-06-03T06:45:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:45:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691904 2024-06-03T06:45:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691904 2024-06-03T06:45:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691904' 2024-06-03T06:45:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:239260: waiting osd.7 seq 171798691904 2024-06-03T06:45:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:45:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691904 -lt 171798691904 2024-06-03T06:45:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528375 2024-06-03T06:45:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:45:57.618 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528375 2024-06-03T06:45:57.618 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:57.618 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528375 2024-06-03T06:45:57.618 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528375' 2024-06-03T06:45:57.618 INFO:tasks.workunit.client.0.smithi039.stderr:239260: waiting osd.8 seq 193273528375 2024-06-03T06:45:57.618 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:45:57.618 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528376 -lt 193273528375 2024-06-03T06:45:58.556 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.3528376 -lt 193273528374 2024-06-03T06:45:58.556 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:58.556 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631501 2024-06-03T06:45:58.557 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:58.557 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:45:58.557 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631501 2024-06-03T06:45:58.557 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:58.557 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631501 2024-06-03T06:45:58.557 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631501' 2024-06-03T06:45:58.557 INFO:tasks.workunit.client.0.smithi039.stderr:239258: waiting osd.9 seq 442381631501 2024-06-03T06:45:58.557 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:45:58.557 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631502 -lt 442381631501 2024-06-03T06:45:58.557 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:45:58.557 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:58.557 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:58.558 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:45:58.558 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:45:58.558 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:45:58.558 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:45:58.558 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:45:58.558 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:45:58.558 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:45:58.558 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:45:58.558 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:45:58.558 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:45:58.558 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:58.558 INFO:tasks.workunit.client.0.smithi039.stderr:239258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:58.559 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:45:58.559 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:45:58.559 INFO:tasks.workunit.client.0.smithi039.stderr:239258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:45:58.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:45:58.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:45:58.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 239257 2024-06-03T06:45:59.014 INFO:tasks.workunit.client.0.smithi039.stderr:sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:45:59.014 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631502 2024-06-03T06:45:59.014 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:45:59.014 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:45:59.014 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631502 2024-06-03T06:45:59.014 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:45:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631502 2024-06-03T06:45:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631502' 2024-06-03T06:45:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:239260: waiting osd.9 seq 442381631502 2024-06-03T06:45:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:45:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631502 -lt 442381631502 2024-06-03T06:45:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:45:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:45:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:45:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:45:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:45:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:45:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:45:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:45:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:45:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:45:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:45:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:45:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:239260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:45:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:45:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:45:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:239260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:45:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:45:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:45:59.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:45:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:45:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-03T06:45:59.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:45:59.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:45:59.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:45:59.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T06:45:59.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:45:59.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:45:59.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:45:59.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:45:59.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:45:59.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:45:59.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:45:59.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:45:59.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:45:59.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:45:59.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:45:59.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:45:59.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:45:59.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:45:59.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:45:59.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:45:59.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:45:59.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:45:59.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:45:59.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:45:59.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:45:59.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:45:59.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:45:59.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:45:59.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:45:59.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:45:59.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:45:59.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:45:59.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T06:45:59.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T06:45:59.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T06:45:59.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T06:45:59.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:45:59.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:45:59.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:45:59.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:45:59.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:45:59.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:45:59.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:45:59.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:45:59.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:45:59.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:45:59.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:45:59.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:45:59.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:45:59.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:45:59.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:45:59.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:45:59.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:45:59.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:45:59.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:45:59.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:45:59.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:45:59.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:45:59.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:45:59.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:45:59.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:45:59.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:45:59.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:45:59.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:45:59.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T06:45:59.255 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:45:59.255 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:45:59.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:45:59.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T06:45:59.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:45:59.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T06:45:59.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:45:59.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T06:45:59.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-06-03T06:45:59.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-06-03T06:45:59.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-06-03T06:45:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-06-03T06:45:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-06-03T06:45:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-06-03T06:45:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-06-03T06:45:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T06:45:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T06:45:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T06:45:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T06:45:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T06:45:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T06:45:59.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T06:45:59.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:45:59.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:45:59.651 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:45:59.651 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:45:59.651 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:45:59.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:45:59.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T06:45:59.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T06:45:59.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T06:45:59.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:45:59.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:45:59.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T06:45:59.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T06:45:59.829 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:45:59.829 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:45:59.829 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:45:59.829 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:45:59.829 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:45:59.829 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:45:59.829 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:45:59.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:45:59.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T06:45:59.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T06:45:59.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:45:59.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:45:59.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T06:45:59.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T06:45:59.913 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:45:59.913 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:45:59.913 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:45:59.914 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:45:59.914 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:45:59.914 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:45:59.914 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:45:59.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:45:59.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T06:45:59.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-06-03T06:45:59.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T06:45:59.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T06:45:59.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T06:45:59.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T06:45:59.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T06:45:59.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T06:46:00.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T06:46:00.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:46:00.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:46:00.812 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:46:00.812 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:46:00.812 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:46:00.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:46:00.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:46:00.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:46:00.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-06-03T06:46:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:46:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-06-03T06:46:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:46:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:46:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T06:46:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:46:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:46:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:46:00.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:46:00.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:46:00.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:46:00.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:46:00.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:46:00.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:46:00.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:46:00.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:46:00.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:46:00.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:46:00.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:46:00.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:46:00.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:46:00.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:46:00.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f91387ea-380d-4487-967e-a7f16a54abe6 2024-06-03T06:46:00.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f91387ea-380d-4487-967e-a7f16a54abe6' 2024-06-03T06:46:00.849 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 f91387ea-380d-4487-967e-a7f16a54abe6 2024-06-03T06:46:00.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:46:00.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCoZl1mN4KHMxAAq2kB63pNNgCZC36zl7Omzg== 2024-06-03T06:46:00.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCoZl1mN4KHMxAAq2kB63pNNgCZC36zl7Omzg=="}' 2024-06-03T06:46:00.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f91387ea-380d-4487-967e-a7f16a54abe6 -i td/osd-scrub-repair/0/new.json 2024-06-03T06:46:01.084 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:46:01.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T06:46:01.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCoZl1mN4KHMxAAq2kB63pNNgCZC36zl7Omzg== --osd-uuid f91387ea-380d-4487-967e-a7f16a54abe6 2024-06-03T06:46:01.125 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:01.124+0000 7f98a3a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:46:01.135 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:01.134+0000 7f98a3a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:46:01.136 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:01.136+0000 7f98a3a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:46:01.137 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:01.136+0000 7f98a3a0d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:46:01.137 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:01.136+0000 7f98a3a0d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:46:01.137 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:01.136+0000 7f98a3a0d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:46:01.219 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:01.218+0000 7f98a3a0d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T06:46:06.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T06:46:06.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:46:06.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T06:46:06.844 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:46:06.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:46:07.233 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:46:07.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T06:46:07.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:46:07.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:46:07.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:46:07.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:46:07.261 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:07.260+0000 7f27a7815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:46:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:07.267+0000 7f27a7815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:46:07.269 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:07.268+0000 7f27a7815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:46:07.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T06:46:07.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:46:07.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:46:07.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:46:07.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:46:07.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:07.557 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:46:07.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:46:07.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:07.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:07.800 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:07.800+0000 7f27a7815640 -1 Falling back to public interface 2024-06-03T06:46:07.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:08.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:08.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:08.848 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:46:08.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:46:08.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:08.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:09.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:10.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:10.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:10.147 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:46:10.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:46:10.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:10.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:10.210 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:10.209+0000 7f27a7815640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:46:10.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:11.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:11.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:11.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:46:11.512 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:46:11.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:11.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:11.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:12.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:12.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:12.824 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:46:12.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:46:12.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:12.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:13.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:14.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:14.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:14.237 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:46:14.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:46:14.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:14.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:14.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:15.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:15.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:15.538 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:46:15.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:46:15.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:15.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:15.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:16.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:16.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:16.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:46:16.842 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:46:16.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:16.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:17.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:18.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:18.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:18.203 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:46:18.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:46:18.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:18.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:18.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:19.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:19.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:19.509 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:46:19.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:46:19.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:19.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:19.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:20.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:20.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:20.832 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:46:20.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:46:20.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:20.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:21.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:22.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:22.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:22.223 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:46:22.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:46:22.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:22.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:22.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:23.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:23.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:23.608 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:46:23.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:46:23.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:23.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:24.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:25.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:25.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:25.303 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:46:25.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:46:25.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:25.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:25.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:26.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:26.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:26.972 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:46:26.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:46:26.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:26.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:27.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:28.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:28.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:28.399 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:46:28.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:46:28.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:28.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:29.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:30.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:30.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:30.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:46:30.110 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:46:30.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:30.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:30.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:31.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:31.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:31.724 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:46:31.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:46:31.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:31.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:32.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:33.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:33.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:33.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:46:33.527 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:46:33.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:33.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:46:33.831 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1522512486,v1:127.0.0.1:6803/1522512486] [v2:127.0.0.1:6804/1522512486,v1:127.0.0.1:6805/1522512486] exists,up f91387ea-380d-4487-967e-a7f16a54abe6 2024-06-03T06:46:33.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:46:33.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:46:33.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:46:33.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:46:33.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-06-03T06:46:33.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:46:33.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:46:33.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T06:46:33.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:46:33.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:46:33.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:46:33.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:46:33.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:46:33.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:46:33.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:46:33.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:46:33.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:46:33.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:46:33.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:46:33.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:46:33.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:46:33.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:46:33.838 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:46:33.838 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:46:33.838 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:46:33.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:46:33.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:46:33.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:46:33.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:46:33.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:46:33.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:46:33.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:46:33.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:46:33.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:46:33.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:46:33.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:46:33.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:46:33.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:46:33.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:46:33.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:46:33.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f3ec3be7-8df4-440d-a1f1-66ae62b9c3c9 2024-06-03T06:46:33.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f3ec3be7-8df4-440d-a1f1-66ae62b9c3c9' 2024-06-03T06:46:33.840 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 f3ec3be7-8df4-440d-a1f1-66ae62b9c3c9 2024-06-03T06:46:33.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:46:33.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJZl1mDlDkMhAA8JfKyUhZ2Vs3dQ9ngsfOVw== 2024-06-03T06:46:33.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJZl1mDlDkMhAA8JfKyUhZ2Vs3dQ9ngsfOVw=="}' 2024-06-03T06:46:33.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f3ec3be7-8df4-440d-a1f1-66ae62b9c3c9 -i td/osd-scrub-repair/1/new.json 2024-06-03T06:46:34.225 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:46:34.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T06:46:34.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDJZl1mDlDkMhAA8JfKyUhZ2Vs3dQ9ngsfOVw== --osd-uuid f3ec3be7-8df4-440d-a1f1-66ae62b9c3c9 2024-06-03T06:46:34.267 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:34.267+0000 7f13ed42e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:46:34.310 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:34.309+0000 7f13ed42e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:46:34.311 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:34.311+0000 7f13ed42e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:46:34.311 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:34.311+0000 7f13ed42e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:46:34.312 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:34.311+0000 7f13ed42e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:46:34.312 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:34.311+0000 7f13ed42e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:46:34.385 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:34.385+0000 7f13ed42e640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T06:46:40.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T06:46:40.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:46:40.775 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:46:40.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T06:46:40.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:46:41.153 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:46:41.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T06:46:41.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:46:41.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:46:41.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:46:41.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:46:41.181 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:41.180+0000 7fa321fda640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:46:41.183 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:41.182+0000 7fa321fda640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:46:41.184 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:41.183+0000 7fa321fda640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:46:41.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T06:46:41.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:46:41.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:46:41.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:46:41.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:46:41.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:41.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:46:41.447 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:46:41.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:41.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:46:41.699 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:41.697+0000 7fa321fda640 -1 Falling back to public interface 2024-06-03T06:46:41.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:42.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:42.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:42.743 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:46:42.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:46:42.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:42.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:46:43.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:44.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:44.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:44.045 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:46:44.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:46:44.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:44.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:46:44.135 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:46:44.133+0000 7fa321fda640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:46:44.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:45.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:45.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:45.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:46:45.353 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:46:45.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:45.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:46:45.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:46.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:46.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:46.654 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:46:46.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:46:46.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:46.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:46:46.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:47.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:47.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:47.973 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:46:47.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:46:47.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:47.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:46:48.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:49.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:49.358 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:46:49.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:49.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:46:49.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:49.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:46:49.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:50.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:50.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:50.663 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:46:50.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:46:50.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:50.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:46:50.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:51.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:51.978 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:46:51.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:51.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:46:51.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:51.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:46:52.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:53.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:53.343 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:46:53.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:53.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:46:53.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:53.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:46:53.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:54.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:54.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:54.646 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:46:54.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:46:54.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:54.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:46:54.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:55.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:55.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:55.983 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:46:55.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:46:55.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:55.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:46:56.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:57.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:57.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:57.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:46:57.297 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:46:57.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:57.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:46:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:46:59.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:46:59.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:46:59.037 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:46:59.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:46:59.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:46:59.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:46:59.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:00.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:00.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:00.351 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:47:00.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:47:00.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:00.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:47:00.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:01.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:01.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:01.874 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:47:01.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:47:01.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:01.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:47:02.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:03.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:03.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:03.585 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:47:03.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:47:03.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:03.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:47:04.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:05.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:05.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:05.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:47:05.254 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:47:05.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:05.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:47:06.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:07.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:07.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:07.124 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:47:07.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:47:07.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:07.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:47:07.435 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1202130659,v1:127.0.0.1:6811/1202130659] [v2:127.0.0.1:6812/1202130659,v1:127.0.0.1:6813/1202130659] exists,up f3ec3be7-8df4-440d-a1f1-66ae62b9c3c9 2024-06-03T06:47:07.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:47:07.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:47:07.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:47:07.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:47:07.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-06-03T06:47:07.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:47:07.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:47:07.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T06:47:07.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:47:07.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:47:07.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:47:07.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:47:07.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:47:07.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:47:07.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:47:07.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:47:07.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:47:07.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:47:07.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:47:07.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:47:07.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:47:07.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:47:07.440 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:47:07.440 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:47:07.440 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:47:07.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:47:07.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:47:07.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:47:07.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:47:07.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:47:07.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:47:07.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:47:07.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:47:07.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:47:07.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:47:07.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:47:07.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:47:07.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:47:07.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:47:07.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:47:07.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bdb305b6-ae18-40db-9431-fcaeba318523 2024-06-03T06:47:07.442 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 bdb305b6-ae18-40db-9431-fcaeba318523 2024-06-03T06:47:07.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bdb305b6-ae18-40db-9431-fcaeba318523' 2024-06-03T06:47:07.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:47:07.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDrZl1mPak7GxAAio0hTSL2olDO8e8QSY93/g== 2024-06-03T06:47:07.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDrZl1mPak7GxAAio0hTSL2olDO8e8QSY93/g=="}' 2024-06-03T06:47:07.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bdb305b6-ae18-40db-9431-fcaeba318523 -i td/osd-scrub-repair/2/new.json 2024-06-03T06:47:07.808 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:47:07.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-03T06:47:07.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDrZl1mPak7GxAAio0hTSL2olDO8e8QSY93/g== --osd-uuid bdb305b6-ae18-40db-9431-fcaeba318523 2024-06-03T06:47:07.850 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:07.849+0000 7f4fa93b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:47:07.893 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:07.893+0000 7f4fa93b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:47:07.894 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:07.894+0000 7f4fa93b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:47:07.895 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:07.894+0000 7f4fa93b8640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:47:07.895 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:07.894+0000 7f4fa93b8640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:47:07.895 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:07.894+0000 7f4fa93b8640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:47:07.968 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:07.967+0000 7f4fa93b8640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T06:47:14.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-03T06:47:14.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:47:14.286 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T06:47:14.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T06:47:14.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:47:14.666 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:47:14.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T06:47:14.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:47:14.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:47:14.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:47:14.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:47:14.694 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:14.693+0000 7f3d0cf31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:47:14.696 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:14.695+0000 7f3d0cf31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:47:14.697 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:14.696+0000 7f3d0cf31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:47:14.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T06:47:14.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:47:14.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:47:14.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:47:14.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:47:14.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:14.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:47:14.955 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:47:14.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:14.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:15.209 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:15.209+0000 7f3d0cf31640 -1 Falling back to public interface 2024-06-03T06:47:15.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:16.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:16.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:16.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:47:16.245 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:47:16.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:16.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:16.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:17.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:17.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:17.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:47:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:47:17.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:17.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:17.577 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:17.577+0000 7f3d0cf31640 -1 osd.2 0 log_to_monitors true 2024-06-03T06:47:17.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:18.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:18.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:18.920 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:47:18.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:47:18.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:18.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:19.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:20.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:20.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:20.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:47:20.276 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:47:20.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:20.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:20.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:21.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:21.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:21.576 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:47:21.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:47:21.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:21.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:21.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:22.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:22.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:22.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:47:22.881 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:47:22.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:22.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:23.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:24.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:24.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:24.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:47:24.191 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:47:24.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:24.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:24.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:25.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:25.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:25.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:47:25.569 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:47:25.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:25.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:25.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:26.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:26.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:26.870 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:47:26.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:47:26.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:26.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:27.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:28.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:28.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:28.179 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:47:28.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:47:28.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:28.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:28.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:29.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:29.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:29.576 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:47:29.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:47:29.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:29.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:29.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:30.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:30.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:30.877 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:47:30.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:47:30.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:30.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:31.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:32.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:32.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:32.522 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:47:32.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:47:32.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:32.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:33.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:34.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:34.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:34.160 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:47:34.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:47:34.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:34.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:34.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:35.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:35.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:35.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:47:35.678 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:47:35.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:35.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:36.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:37.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:37.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:37.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:47:37.270 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:47:37.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:37.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:37.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:38.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:38.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:38.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:47:38.842 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:47:38.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:38.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:39.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:40.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:40.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:40.662 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:47:40.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:47:40.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:40.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:41.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:42.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:42.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:42.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:47:42.059 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:47:42.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:42.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:47:42.366 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/382353093,v1:127.0.0.1:6819/382353093] [v2:127.0.0.1:6820/382353093,v1:127.0.0.1:6821/382353093] exists,up bdb305b6-ae18-40db-9431-fcaeba318523 2024-06-03T06:47:42.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:47:42.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:47:42.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:47:42.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:47:42.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-06-03T06:47:42.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:47:42.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:47:42.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-03T06:47:42.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:47:42.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:47:42.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:47:42.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:47:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:47:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:47:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:47:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:47:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:47:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:47:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:47:42.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:47:42.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:47:42.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:47:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:47:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:47:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:47:42.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:47:42.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:47:42.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:47:42.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:47:42.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:47:42.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:47:42.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:47:42.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:47:42.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:47:42.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:47:42.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:47:42.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:47:42.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:47:42.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:47:42.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:47:42.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=79666bc5-58ed-418d-906d-594dd41a932e 2024-06-03T06:47:42.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 79666bc5-58ed-418d-906d-594dd41a932e' 2024-06-03T06:47:42.373 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 79666bc5-58ed-418d-906d-594dd41a932e 2024-06-03T06:47:42.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:47:42.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAOZ11mIwYnFxAAgVvQSLHUmPzqo8bFh1HN9A== 2024-06-03T06:47:42.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAOZ11mIwYnFxAAgVvQSLHUmPzqo8bFh1HN9A=="}' 2024-06-03T06:47:42.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 79666bc5-58ed-418d-906d-594dd41a932e -i td/osd-scrub-repair/3/new.json 2024-06-03T06:47:42.757 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:47:42.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-06-03T06:47:42.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAOZ11mIwYnFxAAgVvQSLHUmPzqo8bFh1HN9A== --osd-uuid 79666bc5-58ed-418d-906d-594dd41a932e 2024-06-03T06:47:42.798 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:42.798+0000 7f3bfe22c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:47:42.857 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:42.857+0000 7f3bfe22c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:47:42.858 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:42.858+0000 7f3bfe22c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:47:42.859 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:42.858+0000 7f3bfe22c640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-03T06:47:42.859 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:42.858+0000 7f3bfe22c640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-03T06:47:42.859 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:42.858+0000 7f3bfe22c640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-03T06:47:42.932 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:42.932+0000 7f3bfe22c640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-06-03T06:47:48.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-06-03T06:47:48.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:47:48.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-03T06:47:48.993 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-06-03T06:47:48.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:47:49.376 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-06-03T06:47:49.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-03T06:47:49.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:47:49.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:47:49.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:47:49.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:47:49.404 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:49.403+0000 7fe078c17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:47:49.406 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:49.406+0000 7fe078c17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:47:49.407 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:49.407+0000 7fe078c17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:47:49.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-03T06:47:49.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:47:49.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:47:49.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:47:49.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:47:49.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:49.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:47:49.668 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:47:49.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:49.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:47:49.926 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:49.925+0000 7fe078c17640 -1 Falling back to public interface 2024-06-03T06:47:49.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:50.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:50.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:50.965 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:47:50.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:47:50.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:50.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:47:51.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:52.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:52.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:52.264 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:47:52.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:47:52.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:52.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:47:52.395 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:47:52.394+0000 7fe078c17640 -1 osd.3 0 log_to_monitors true 2024-06-03T06:47:52.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:53.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:53.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:53.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:47:53.622 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:47:53.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:53.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:47:54.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:55.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:55.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:55.002 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:47:55.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:47:55.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:55.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:47:55.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:56.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:56.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:56.307 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:47:56.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:47:56.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:56.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:47:56.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:57.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:57.615 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:47:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:47:57.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:47:57.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:47:58.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:47:58.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:47:58.915 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:47:58.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:47:58.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:47:58.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:47:59.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:00.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:00.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:00.257 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:48:00.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:48:00.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:00.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:48:00.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:01.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:01.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:01.704 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:48:01.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:48:01.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:01.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:48:02.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:03.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:03.013 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:48:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:48:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:48:03.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:04.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:04.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:04.429 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:48:04.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:48:04.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:04.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:48:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:05.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:05.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:05.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:48:05.765 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:48:05.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:05.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:48:06.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:07.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:07.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:07.070 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:48:07.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:48:07.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:07.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:48:07.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:08.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:08.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:08.690 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:48:08.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:48:08.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:08.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:48:09.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:10.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:10.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:10.077 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:48:10.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:48:10.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:10.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:48:10.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:11.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:11.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:11.712 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:48:11.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:48:11.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:11.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:48:12.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:13.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:13.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:13.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:48:13.507 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:48:13.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:13.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:48:14.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:15.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:15.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:15.064 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:48:15.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:48:15.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:15.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:48:15.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:16.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:16.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:16.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:48:16.630 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:48:16.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:16.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:48:16.962 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1992599287,v1:127.0.0.1:6827/1992599287] [v2:127.0.0.1:6828/1992599287,v1:127.0.0.1:6829/1992599287] exists,up 79666bc5-58ed-418d-906d-594dd41a932e 2024-06-03T06:48:16.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:48:16.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:48:16.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:48:16.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:48:16.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-06-03T06:48:16.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:48:16.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:48:16.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-06-03T06:48:16.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:48:16.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-06-03T06:48:16.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:48:16.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:48:16.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:48:16.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:48:16.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-06-03T06:48:16.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-06-03T06:48:16.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:48:16.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:48:16.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:48:16.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:48:16.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:48:16.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:48:16.968 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:48:16.968 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:48:16.968 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:48:16.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:48:16.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:48:16.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:48:16.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:48:16.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:48:16.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:48:16.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:48:16.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:48:16.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:48:16.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:48:16.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:48:16.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:48:16.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:48:16.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-06-03T06:48:16.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:48:16.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62d9b267-efc0-4290-bf42-ca99e137cbce 2024-06-03T06:48:16.970 INFO:tasks.workunit.client.0.smithi039.stdout:add osd4 62d9b267-efc0-4290-bf42-ca99e137cbce 2024-06-03T06:48:16.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 62d9b267-efc0-4290-bf42-ca99e137cbce' 2024-06-03T06:48:16.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:48:16.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAwZ11mT5/MOhAAJGkPTis4yT2VJpa2nmXcQg== 2024-06-03T06:48:16.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAwZ11mT5/MOhAAJGkPTis4yT2VJpa2nmXcQg=="}' 2024-06-03T06:48:16.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62d9b267-efc0-4290-bf42-ca99e137cbce -i td/osd-scrub-repair/4/new.json 2024-06-03T06:48:17.332 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:48:17.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-06-03T06:48:17.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAwZ11mT5/MOhAAJGkPTis4yT2VJpa2nmXcQg== --osd-uuid 62d9b267-efc0-4290-bf42-ca99e137cbce 2024-06-03T06:48:17.374 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:17.374+0000 7f9281101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:48:17.417 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:17.417+0000 7f9281101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:48:17.418 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:17.418+0000 7f9281101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:48:17.418 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:17.418+0000 7f9281101640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-03T06:48:17.419 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:17.418+0000 7f9281101640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-03T06:48:17.419 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:17.418+0000 7f9281101640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-03T06:48:17.492 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:17.492+0000 7f9281101640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-06-03T06:48:23.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-06-03T06:48:23.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:48:23.453 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd4 key to auth repository 2024-06-03T06:48:23.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-06-03T06:48:23.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:48:23.829 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.4 2024-06-03T06:48:23.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-06-03T06:48:23.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:48:23.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:48:23.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:48:23.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:48:23.856 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:23.855+0000 7f5f89012640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:48:23.858 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:23.857+0000 7f5f89012640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:48:23.859 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:23.858+0000 7f5f89012640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:48:24.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-06-03T06:48:24.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:48:24.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-06-03T06:48:24.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:48:24.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:48:24.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:24.138 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:48:24.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:48:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:24.382 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:24.381+0000 7f5f89012640 -1 Falling back to public interface 2024-06-03T06:48:24.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:25.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:25.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:25.434 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:48:25.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:48:25.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:25.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:25.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:26.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:26.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:26.737 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:48:26.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:48:26.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:26.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:26.788 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:26.786+0000 7f5f89012640 -1 osd.4 0 log_to_monitors true 2024-06-03T06:48:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:28.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:28.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:28.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:48:28.040 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:48:28.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:28.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:28.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:29.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:29.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:29.445 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:48:29.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:48:29.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:29.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:29.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:30.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:30.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:30.812 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:48:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:48:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:31.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:32.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:32.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:32.119 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:48:32.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:48:32.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:32.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:32.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:33.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:33.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:33.440 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:48:33.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:48:33.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:33.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:33.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:34.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:34.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:34.816 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:48:34.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:48:34.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:34.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:35.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:36.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:36.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:36.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:48:36.118 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:48:36.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:36.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:36.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:37.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:37.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:37.460 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:48:37.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:48:37.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:37.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:37.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:38.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:38.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:38.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:48:38.800 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:48:38.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:38.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:39.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:40.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:40.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:40.102 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:48:40.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:48:40.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:40.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:40.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:41.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:41.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:41.704 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:48:41.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:48:41.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:41.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:42.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:43.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:43.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:43.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:48:43.249 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:48:43.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:43.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:44.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:45.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:45.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:45.073 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:48:45.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:48:45.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:45.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:45.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:46.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:46.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:46.865 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:48:46.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:48:46.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:46.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:47.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:48.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:48.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:48.542 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:48:48.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:48:48.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:48.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:49.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:50.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:50.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:50.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:48:50.055 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:48:50.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:50.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:50.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:48:51.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:48:51.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:51.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:48:51.467 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:48:51.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:51.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-03T06:48:51.767 INFO:tasks.workunit.client.0.smithi039.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1063687366,v1:127.0.0.1:6835/1063687366] [v2:127.0.0.1:6836/1063687366,v1:127.0.0.1:6837/1063687366] exists,up 62d9b267-efc0-4290-bf42-ca99e137cbce 2024-06-03T06:48:51.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:48:51.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:48:51.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:48:51.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:48:51.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-06-03T06:48:51.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:48:51.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:48:51.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-06-03T06:48:51.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:48:51.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:48:51.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:48:51.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:48:51.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:48:51.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:48:51.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-03T06:48:51.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-03T06:48:51.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:48:51.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:48:51.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:48:51.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:48:51.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:48:51.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:48:51.773 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:48:51.773 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:48:51.773 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:48:51.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:48:51.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:48:51.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:48:51.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:48:51.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:48:51.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:48:51.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:48:51.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:48:51.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:48:51.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:48:51.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:48:51.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:48:51.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:48:51.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-06-03T06:48:51.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:48:51.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=915fce46-b35c-47e4-a164-8e162d251809 2024-06-03T06:48:51.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 915fce46-b35c-47e4-a164-8e162d251809' 2024-06-03T06:48:51.775 INFO:tasks.workunit.client.0.smithi039.stdout:add osd5 915fce46-b35c-47e4-a164-8e162d251809 2024-06-03T06:48:51.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:48:51.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTZ11mIlslLxAANyn0aArcyAu46bn/vLHZUQ== 2024-06-03T06:48:51.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTZ11mIlslLxAANyn0aArcyAu46bn/vLHZUQ=="}' 2024-06-03T06:48:51.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 915fce46-b35c-47e4-a164-8e162d251809 -i td/osd-scrub-repair/5/new.json 2024-06-03T06:48:52.227 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:48:52.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-06-03T06:48:52.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTZ11mIlslLxAANyn0aArcyAu46bn/vLHZUQ== --osd-uuid 915fce46-b35c-47e4-a164-8e162d251809 2024-06-03T06:48:52.269 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:52.268+0000 7fdd80230640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:48:52.311 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:52.311+0000 7fdd80230640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:48:52.312 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:52.312+0000 7fdd80230640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:48:52.313 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:52.312+0000 7fdd80230640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-03T06:48:52.313 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:52.312+0000 7fdd80230640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-03T06:48:52.313 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:52.312+0000 7fdd80230640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-03T06:48:52.386 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:52.386+0000 7fdd80230640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-06-03T06:48:58.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-06-03T06:48:58.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:48:58.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-06-03T06:48:58.723 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd5 key to auth repository 2024-06-03T06:48:58.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:48:59.099 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.5 2024-06-03T06:48:59.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-06-03T06:48:59.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:48:59.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:48:59.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:48:59.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:48:59.127 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:59.127+0000 7efd46a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:48:59.129 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:59.129+0000 7efd46a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:48:59.130 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:59.130+0000 7efd46a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:48:59.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-06-03T06:48:59.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:48:59.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-03T06:48:59.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:48:59.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:48:59.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:48:59.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:48:59.387 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:48:59.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:48:59.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:48:59.645 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:48:59.645+0000 7efd46a11640 -1 Falling back to public interface 2024-06-03T06:48:59.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:00.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:00.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:00.676 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:49:00.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:49:00.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:00.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:00.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:01.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:01.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:01.977 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:49:01.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:49:01.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:01.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:02.058 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:49:02.058+0000 7efd46a11640 -1 osd.5 0 log_to_monitors true 2024-06-03T06:49:02.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:03.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:03.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:03.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:49:03.310 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:49:03.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:03.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:03.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:04.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:04.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:04.610 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:49:04.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:49:04.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:04.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:04.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:05.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:05.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:05.950 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:49:05.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:49:05.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:05.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:06.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:07.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:07.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:07.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:49:07.252 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:49:07.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:07.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:07.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:08.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:08.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:08.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:49:08.553 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:49:08.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:08.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:08.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:09.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:09.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:09.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:49:09.917 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:49:09.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:09.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:10.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:11.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:11.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:11.223 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:49:11.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:49:11.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:11.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:11.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:12.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:12.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:12.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:49:12.529 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:49:12.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:12.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:12.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:13.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:13.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:13.902 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:49:13.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:49:13.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:13.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:14.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:15.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:15.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:15.268 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:49:15.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:49:15.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:15.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:15.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:16.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:16.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:16.801 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:49:16.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:49:16.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:16.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:17.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:18.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:18.279 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:49:18.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:18.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:49:18.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:18.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:18.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:19.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:19.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:19.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:49:19.649 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:49:19.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:19.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:20.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:21.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:21.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:21.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:49:21.335 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:49:21.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:21.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:22.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:23.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:23.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:23.138 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:49:23.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:49:23.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:23.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:23.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:24.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:24.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:24.959 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:49:24.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:49:24.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:24.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:25.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:26.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:26.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:26.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:49:26.355 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:49:26.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:26.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:49:26.660 INFO:tasks.workunit.client.0.smithi039.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2446460070,v1:127.0.0.1:6843/2446460070] [v2:127.0.0.1:6844/2446460070,v1:127.0.0.1:6845/2446460070] exists,up 915fce46-b35c-47e4-a164-8e162d251809 2024-06-03T06:49:26.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:49:26.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:49:26.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:49:26.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:49:26.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-06-03T06:49:26.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:49:26.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:49:26.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-06-03T06:49:26.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:49:26.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-06-03T06:49:26.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:49:26.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:49:26.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:49:26.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:49:26.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-06-03T06:49:26.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-06-03T06:49:26.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:49:26.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:49:26.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:49:26.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:49:26.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:49:26.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:49:26.666 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:49:26.666 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:49:26.666 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:49:26.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:49:26.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:49:26.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:49:26.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:49:26.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:49:26.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:49:26.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:49:26.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:49:26.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:49:26.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:49:26.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:49:26.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:49:26.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:49:26.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-06-03T06:49:26.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:49:26.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ac7f228e-2310-4f92-9fef-4a5154f2d67b 2024-06-03T06:49:26.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 ac7f228e-2310-4f92-9fef-4a5154f2d67b' 2024-06-03T06:49:26.667 INFO:tasks.workunit.client.0.smithi039.stdout:add osd6 ac7f228e-2310-4f92-9fef-4a5154f2d67b 2024-06-03T06:49:26.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:49:26.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB2Z11maS+tKBAAV0p8inczoPqcegQQ4waNAQ== 2024-06-03T06:49:26.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB2Z11maS+tKBAAV0p8inczoPqcegQQ4waNAQ=="}' 2024-06-03T06:49:26.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ac7f228e-2310-4f92-9fef-4a5154f2d67b -i td/osd-scrub-repair/6/new.json 2024-06-03T06:49:27.140 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:49:27.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-06-03T06:49:27.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB2Z11maS+tKBAAV0p8inczoPqcegQQ4waNAQ== --osd-uuid ac7f228e-2310-4f92-9fef-4a5154f2d67b 2024-06-03T06:49:27.182 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:49:27.182+0000 7fd24bb99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:49:27.233 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:49:27.233+0000 7fd24bb99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:49:27.234 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:49:27.234+0000 7fd24bb99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:49:27.234 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:49:27.234+0000 7fd24bb99640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-03T06:49:27.234 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:49:27.234+0000 7fd24bb99640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-03T06:49:27.235 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:49:27.234+0000 7fd24bb99640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-03T06:49:27.308 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:49:27.307+0000 7fd24bb99640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-06-03T06:49:33.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-06-03T06:49:33.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:49:33.643 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd6 key to auth repository 2024-06-03T06:49:33.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-06-03T06:49:33.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:49:34.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-06-03T06:49:34.046 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.6 2024-06-03T06:49:34.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:49:34.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:49:34.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:49:34.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:49:34.073 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:49:34.073+0000 7f92c480d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:49:34.075 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:49:34.075+0000 7f92c480d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:49:34.076 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:49:34.076+0000 7f92c480d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:49:34.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-06-03T06:49:34.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:49:34.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-06-03T06:49:34.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:49:34.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:49:34.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:34.338 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:49:34.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:49:34.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:34.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:34.591 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:49:34.591+0000 7f92c480d640 -1 Falling back to public interface 2024-06-03T06:49:34.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:35.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:35.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:35.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:49:35.632 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:49:35.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:35.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:36.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:36.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:36.933 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:49:36.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:49:36.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:36.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:36.961 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:49:36.961+0000 7f92c480d640 -1 osd.6 0 log_to_monitors true 2024-06-03T06:49:37.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:38.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:38.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:38.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:49:38.334 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:49:38.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:38.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:38.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:39.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:39.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:39.727 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:49:39.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:49:39.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:39.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:40.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:41.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:41.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:41.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:49:41.125 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:49:41.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:41.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:41.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:42.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:42.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:42.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:49:42.429 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:49:42.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:42.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:42.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:43.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:43.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:43.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:49:43.730 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:49:43.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:43.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:44.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:45.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:45.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:45.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:49:45.189 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:49:45.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:45.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:45.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:46.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:46.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:46.492 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:49:46.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:49:46.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:46.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:46.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:47.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:47.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:47.796 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:49:47.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:49:47.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:47.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:48.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:49.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:49.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:49.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:49:49.187 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:49:49.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:49.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:49.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:50.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:50.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:50.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:49:50.489 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:49:50.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:50.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:51.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:52.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:52.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:52.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:49:52.123 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:49:52.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:52.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:52.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:53.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:53.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:53.777 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:49:53.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:49:53.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:53.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:54.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:55.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:55.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:55.591 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:49:55.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:49:55.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:55.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:56.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:57.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:57.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:57.266 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:49:57.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:49:57.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:57.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:57.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:49:58.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:49:58.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:49:58.897 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:49:58.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:49:58.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:49:58.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:49:59.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:00.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:00.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:00.937 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:50:00.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:50:00.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:00.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:50:01.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:02.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:02.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:02.264 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:50:02.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:50:02.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:02.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-03T06:50:02.561 INFO:tasks.workunit.client.0.smithi039.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/96337712,v1:127.0.0.1:6851/96337712] [v2:127.0.0.1:6852/96337712,v1:127.0.0.1:6853/96337712] exists,up ac7f228e-2310-4f92-9fef-4a5154f2d67b 2024-06-03T06:50:02.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:50:02.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:50:02.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:50:02.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:50:02.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-06-03T06:50:02.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:50:02.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:50:02.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-06-03T06:50:02.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:50:02.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-06-03T06:50:02.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:50:02.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:50:02.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:50:02.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:50:02.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-06-03T06:50:02.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-06-03T06:50:02.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:50:02.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:50:02.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:50:02.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:50:02.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:50:02.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:50:02.567 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:50:02.567 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:50:02.567 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:50:02.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:50:02.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:50:02.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:50:02.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:50:02.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:50:02.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:50:02.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:50:02.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:50:02.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:50:02.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:50:02.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:50:02.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:50:02.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:50:02.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-06-03T06:50:02.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:50:02.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=438d722c-a5ea-4269-8740-2f0e4a3224f9 2024-06-03T06:50:02.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 438d722c-a5ea-4269-8740-2f0e4a3224f9' 2024-06-03T06:50:02.569 INFO:tasks.workunit.client.0.smithi039.stdout:add osd7 438d722c-a5ea-4269-8740-2f0e4a3224f9 2024-06-03T06:50:02.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:50:02.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCaZ11mxxLWIhAAxR3J1J4+acOxKf4DcI9gsw== 2024-06-03T06:50:02.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCaZ11mxxLWIhAAxR3J1J4+acOxKf4DcI9gsw=="}' 2024-06-03T06:50:02.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 438d722c-a5ea-4269-8740-2f0e4a3224f9 -i td/osd-scrub-repair/7/new.json 2024-06-03T06:50:02.966 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:50:02.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-06-03T06:50:02.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCaZ11mxxLWIhAAxR3J1J4+acOxKf4DcI9gsw== --osd-uuid 438d722c-a5ea-4269-8740-2f0e4a3224f9 2024-06-03T06:50:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:03.007+0000 7f58c85b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:50:03.076 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:03.076+0000 7f58c85b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:50:03.077 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:03.077+0000 7f58c85b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:50:03.078 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:03.077+0000 7f58c85b8640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-03T06:50:03.078 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:03.077+0000 7f58c85b8640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-03T06:50:03.078 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:03.077+0000 7f58c85b8640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-03T06:50:03.176 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:03.176+0000 7f58c85b8640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-06-03T06:50:09.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-06-03T06:50:09.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:50:09.237 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd7 key to auth repository 2024-06-03T06:50:09.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-06-03T06:50:09.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:50:09.612 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.7 2024-06-03T06:50:09.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-06-03T06:50:09.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:50:09.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:50:09.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:50:09.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:50:09.639 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:09.638+0000 7fad93a26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:50:09.641 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:09.640+0000 7fad93a26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:50:09.642 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:09.641+0000 7fad93a26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:50:09.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-06-03T06:50:09.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:50:09.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-06-03T06:50:09.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:50:09.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:50:09.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:09.903 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:50:09.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:50:09.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:09.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:10.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:10.190 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:10.189+0000 7fad93a26640 -1 Falling back to public interface 2024-06-03T06:50:11.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:11.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:11.191 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:50:11.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:50:11.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:11.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:11.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:12.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:12.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:12.494 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:50:12.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:50:12.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:12.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:12.521 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:12.519+0000 7fad93a26640 -1 osd.7 0 log_to_monitors true 2024-06-03T06:50:12.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:13.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:13.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:13.853 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:50:13.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:50:13.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:13.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:14.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:15.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:15.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:15.239 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:50:15.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:50:15.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:15.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:15.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:16.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:16.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:16.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:50:16.642 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:50:16.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:16.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:16.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:17.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:17.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:17.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:50:17.943 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:50:17.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:17.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:18.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:19.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:19.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:19.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:50:19.266 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:50:19.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:19.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:19.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:20.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:20.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:20.615 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:50:20.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:50:20.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:20.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:20.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:21.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:21.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:21.917 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:50:21.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:50:21.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:21.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:22.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:23.219 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:50:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:50:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:23.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:24.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:24.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:24.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:50:24.518 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:50:24.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:24.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:24.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:25.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:25.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:25.823 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:50:25.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:50:25.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:25.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:26.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:27.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:27.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:27.273 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:50:27.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:50:27.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:27.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:27.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:28.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:28.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:28.966 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:50:28.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:50:28.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:28.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:29.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:30.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:30.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:30.881 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:50:30.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:50:30.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:30.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:31.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:32.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:32.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:32.288 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:50:32.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:50:32.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:32.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:32.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:33.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:33.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:33.912 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:50:33.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:50:33.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:33.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:34.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:35.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:35.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:35.466 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:50:35.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:50:35.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:35.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-03T06:50:35.830 INFO:tasks.workunit.client.0.smithi039.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1482582526,v1:127.0.0.1:6859/1482582526] [v2:127.0.0.1:6860/1482582526,v1:127.0.0.1:6861/1482582526] exists,up 438d722c-a5ea-4269-8740-2f0e4a3224f9 2024-06-03T06:50:35.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:50:35.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:50:35.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:50:35.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:50:35.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-06-03T06:50:35.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:50:35.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:50:35.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-06-03T06:50:35.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:50:35.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-06-03T06:50:35.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:50:35.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:50:35.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:50:35.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:50:35.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-06-03T06:50:35.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-06-03T06:50:35.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:50:35.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:50:35.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:50:35.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:50:35.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:50:35.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:50:35.836 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:50:35.836 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:50:35.836 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:50:35.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:50:35.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:50:35.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:50:35.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:50:35.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:50:35.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:50:35.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:50:35.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:50:35.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:50:35.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:50:35.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:50:35.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:50:35.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:50:35.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-06-03T06:50:35.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:50:35.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aa91cd61-2534-4a44-a560-f26e1ea176c7 2024-06-03T06:50:35.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 aa91cd61-2534-4a44-a560-f26e1ea176c7' 2024-06-03T06:50:35.838 INFO:tasks.workunit.client.0.smithi039.stdout:add osd8 aa91cd61-2534-4a44-a560-f26e1ea176c7 2024-06-03T06:50:35.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:50:35.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC7Z11mY0bbMhAAvaPxhc0SV6er52bRFA83Pg== 2024-06-03T06:50:35.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC7Z11mY0bbMhAAvaPxhc0SV6er52bRFA83Pg=="}' 2024-06-03T06:50:35.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aa91cd61-2534-4a44-a560-f26e1ea176c7 -i td/osd-scrub-repair/8/new.json 2024-06-03T06:50:36.236 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:50:36.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-06-03T06:50:36.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC7Z11mY0bbMhAAvaPxhc0SV6er52bRFA83Pg== --osd-uuid aa91cd61-2534-4a44-a560-f26e1ea176c7 2024-06-03T06:50:36.278 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:36.278+0000 7fa2c901f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:50:36.296 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:36.296+0000 7fa2c901f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:50:36.297 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:36.297+0000 7fa2c901f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:50:36.298 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:36.298+0000 7fa2c901f640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-03T06:50:36.298 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:36.298+0000 7fa2c901f640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-03T06:50:36.298 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:36.298+0000 7fa2c901f640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-03T06:50:36.429 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:36.429+0000 7fa2c901f640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-06-03T06:50:42.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-06-03T06:50:42.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:50:42.691 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd8 key to auth repository 2024-06-03T06:50:42.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-06-03T06:50:42.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:50:43.066 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.8 2024-06-03T06:50:43.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-06-03T06:50:43.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:50:43.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:50:43.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:50:43.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:50:43.094 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:43.093+0000 7f997056e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:50:43.096 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:43.096+0000 7f997056e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:50:43.097 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:43.097+0000 7f997056e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:50:43.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-06-03T06:50:43.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:50:43.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-06-03T06:50:43.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:50:43.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:50:43.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:43.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:50:43.358 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:50:43.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:43.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:50:43.612 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:43.612+0000 7f997056e640 -1 Falling back to public interface 2024-06-03T06:50:43.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:44.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:44.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:44.646 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:50:44.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:50:44.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:44.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:50:44.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:45.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:45.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:45.946 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:50:45.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:50:45.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:45.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:50:46.072 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:50:46.072+0000 7f997056e640 -1 osd.8 0 log_to_monitors true 2024-06-03T06:50:46.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:47.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:47.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:47.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:50:47.342 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:50:47.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:47.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:50:47.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:48.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:48.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:48.645 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:50:48.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:50:48.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:48.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:50:49.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:50.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:50.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:50.006 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:50:50.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:50:50.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:50.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:50:50.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:51.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:51.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:51.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:50:51.338 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:50:51.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:51.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:50:51.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:52.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:52.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:52.639 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:50:52.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:50:52.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:52.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:50:52.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:53.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:53.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:53.949 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:50:53.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:50:53.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:53.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:50:54.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:55.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:55.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:55.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:50:55.314 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:50:55.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:55.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:50:55.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:56.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:56.616 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:50:56.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:56.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:50:56.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:56.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:50:56.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:57.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:57.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:57.960 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:50:57.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:50:57.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:57.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:50:58.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:50:59.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:50:59.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:50:59.298 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:50:59.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:50:59.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:50:59.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:50:59.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:00.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:00.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:00.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:51:00.829 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:51:00.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:00.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:51:01.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:02.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:02.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:02.343 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:51:02.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:51:02.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:02.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:51:02.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:03.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:03.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:03.788 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:51:03.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:51:03.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:03.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:51:04.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:05.445 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:51:05.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:51:05.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:05.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:51:06.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:07.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:07.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:07.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:51:07.154 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:51:07.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:07.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:51:07.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:08.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:08.854 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:51:08.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:08.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:51:08.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:08.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:51:09.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:10.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:10.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:10.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:51:10.214 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:51:10.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:10.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-03T06:51:10.541 INFO:tasks.workunit.client.0.smithi039.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/4282188765,v1:127.0.0.1:6867/4282188765] [v2:127.0.0.1:6868/4282188765,v1:127.0.0.1:6869/4282188765] exists,up aa91cd61-2534-4a44-a560-f26e1ea176c7 2024-06-03T06:51:10.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:51:10.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:51:10.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:51:10.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-03T06:51:10.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-06-03T06:51:10.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:51:10.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:51:10.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-06-03T06:51:10.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:51:10.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:51:10.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:51:10.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:51:10.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:51:10.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:51:10.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-03T06:51:10.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-03T06:51:10.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:51:10.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:51:10.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:51:10.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:51:10.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:51:10.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:51:10.547 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:51:10.547 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:51:10.548 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:51:10.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:51:10.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:51:10.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:51:10.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:51:10.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:51:10.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:51:10.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:51:10.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:51:10.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:51:10.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:51:10.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:51:10.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:51:10.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:51:10.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-06-03T06:51:10.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:51:10.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fc630115-1953-4e3d-885f-d2824414ef72 2024-06-03T06:51:10.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 fc630115-1953-4e3d-885f-d2824414ef72' 2024-06-03T06:51:10.549 INFO:tasks.workunit.client.0.smithi039.stdout:add osd9 fc630115-1953-4e3d-885f-d2824414ef72 2024-06-03T06:51:10.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:51:10.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDeZ11meW6hIRAAiGDdpoYI5nMOyLAHmtjpyw== 2024-06-03T06:51:10.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDeZ11meW6hIRAAiGDdpoYI5nMOyLAHmtjpyw=="}' 2024-06-03T06:51:10.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fc630115-1953-4e3d-885f-d2824414ef72 -i td/osd-scrub-repair/9/new.json 2024-06-03T06:51:10.938 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:51:10.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-06-03T06:51:10.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDeZ11meW6hIRAAiGDdpoYI5nMOyLAHmtjpyw== --osd-uuid fc630115-1953-4e3d-885f-d2824414ef72 2024-06-03T06:51:10.980 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:51:10.980+0000 7faa0b20e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:51:11.027 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:51:11.027+0000 7faa0b20e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:51:11.028 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:51:11.028+0000 7faa0b20e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:51:11.028 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:51:11.028+0000 7faa0b20e640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-03T06:51:11.029 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:51:11.028+0000 7faa0b20e640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-03T06:51:11.029 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:51:11.028+0000 7faa0b20e640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-03T06:51:11.106 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:51:11.105+0000 7faa0b20e640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-06-03T06:51:17.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-06-03T06:51:17.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:51:17.362 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd9 key to auth repository 2024-06-03T06:51:17.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-06-03T06:51:17.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:51:17.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-06-03T06:51:17.737 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.9 2024-06-03T06:51:17.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:51:17.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:51:17.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:51:17.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:51:17.765 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:51:17.765+0000 7f6cd61da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:51:17.767 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:51:17.767+0000 7f6cd61da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:51:17.768 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:51:17.768+0000 7f6cd61da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:51:18.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-06-03T06:51:18.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:51:18.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-03T06:51:18.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:51:18.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:51:18.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:18.042 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:51:18.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:51:18.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:18.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:18.287 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:51:18.287+0000 7f6cd61da640 -1 Falling back to public interface 2024-06-03T06:51:18.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:19.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:19.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:19.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:51:19.333 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:51:19.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:19.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:19.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:20.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:20.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:20.634 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:51:20.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:51:20.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:20.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:20.780 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:51:20.780+0000 7f6cd61da640 -1 osd.9 0 log_to_monitors true 2024-06-03T06:51:21.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:22.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:22.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:22.007 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:51:22.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:51:22.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:22.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:22.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:23.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:23.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:23.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:51:23.523 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:51:23.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:23.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:23.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:24.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:24.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:24.857 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:51:24.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:51:24.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:24.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:25.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:26.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:26.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:26.160 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:51:26.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:51:26.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:26.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:26.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:27.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:27.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:27.463 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:51:27.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:51:27.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:27.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:27.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:28.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:28.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:28.866 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:51:28.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:51:28.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:28.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:29.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:30.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:30.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:30.164 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:51:30.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:51:30.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:30.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:30.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:31.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:31.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:31.495 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:51:31.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:51:31.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:31.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:31.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:32.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:32.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:32.843 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:51:32.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:51:32.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:32.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:33.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:34.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:34.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:34.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:51:34.147 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:51:34.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:34.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:34.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:35.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:35.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:35.848 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:51:35.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:51:35.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:35.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:36.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:37.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:37.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:37.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:51:37.484 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:51:37.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:37.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:38.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:39.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:39.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:39.164 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:51:39.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:51:39.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:39.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:39.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:40.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:40.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:40.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:51:40.808 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:51:40.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:40.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:41.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:42.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:42.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:42.278 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:51:42.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:51:42.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:42.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:42.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:43.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:43.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:43.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:51:43.807 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:51:43.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:43.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:44.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:51:45.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:51:45.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:51:45.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:51:45.269 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:51:45.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:51:45.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:51:45.570 INFO:tasks.workunit.client.0.smithi039.stdout:osd.9 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/1882354255,v1:127.0.0.1:6875/1882354255] [v2:127.0.0.1:6876/1882354255,v1:127.0.0.1:6877/1882354255] exists,up fc630115-1953-4e3d-885f-d2824414ef72 2024-06-03T06:51:45.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:51:45.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:51:45.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:51:45.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-06-03T06:51:45.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T06:51:45.854 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-06-03T06:51:45.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T06:51:45.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T06:51:46.258 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T06:51:46.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:51:47.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T06:51:50.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-06-03T06:51:50.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:51:50.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:51:50.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:51:50.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:51:50.318 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:51:50.318 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:51:50.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:51:50.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:51:50.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:51:50.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:51:50.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:51:50.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:51:50.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:51:50.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:51:50.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:51:50.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:51:50.831 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:51:50.831 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:51:50.831 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:51:50.831 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-06-03T06:51:50.831 INFO:tasks.workunit.client.0.smithi039.stderr:5 2024-06-03T06:51:50.831 INFO:tasks.workunit.client.0.smithi039.stderr:6 2024-06-03T06:51:50.831 INFO:tasks.workunit.client.0.smithi039.stderr:7 2024-06-03T06:51:50.831 INFO:tasks.workunit.client.0.smithi039.stderr:8 2024-06-03T06:51:50.831 INFO:tasks.workunit.client.0.smithi039.stderr:9' 2024-06-03T06:51:50.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:51:50.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:51:50.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:51:50.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836550 2024-06-03T06:51:50.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836550 2024-06-03T06:51:50.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550' 2024-06-03T06:51:50.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:51:50.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:51:51.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673022 2024-06-03T06:51:51.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673022 2024-06-03T06:51:51.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022' 2024-06-03T06:51:51.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:51:51.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:51:51.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509496 2024-06-03T06:51:51.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509496 2024-06-03T06:51:51.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509496' 2024-06-03T06:51:51.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:51:51.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:51:51.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345967 2024-06-03T06:51:51.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345967 2024-06-03T06:51:51.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509496 3-85899345967' 2024-06-03T06:51:51.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:51:51.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:51:51.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182440 2024-06-03T06:51:51.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182440 2024-06-03T06:51:51.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509496 3-85899345967 4-107374182440' 2024-06-03T06:51:51.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:51:51.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:51:51.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018913 2024-06-03T06:51:51.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018913 2024-06-03T06:51:51.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509496 3-85899345967 4-107374182440 5-128849018913' 2024-06-03T06:51:51.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:51:51.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:51:51.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855389 2024-06-03T06:51:51.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855389 2024-06-03T06:51:51.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509496 3-85899345967 4-107374182440 5-128849018913 6-150323855389' 2024-06-03T06:51:51.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:51:51.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:51:51.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691860 2024-06-03T06:51:51.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691860 2024-06-03T06:51:51.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509496 3-85899345967 4-107374182440 5-128849018913 6-150323855389 7-171798691860' 2024-06-03T06:51:51.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:51:51.698 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:51:51.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528339 2024-06-03T06:51:51.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528339 2024-06-03T06:51:51.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509496 3-85899345967 4-107374182440 5-128849018913 6-150323855389 7-171798691860 8-193273528339' 2024-06-03T06:51:51.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:51:51.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:51:51.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397510 2024-06-03T06:51:51.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397510 2024-06-03T06:51:51.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509496 3-85899345967 4-107374182440 5-128849018913 6-150323855389 7-171798691860 8-193273528339 9-210453397510' 2024-06-03T06:51:51.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:51:51.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836550 2024-06-03T06:51:51.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:51:51.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:51:51.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836550 2024-06-03T06:51:51.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:51:51.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836550 2024-06-03T06:51:51.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-06-03T06:51:51.918 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836550 2024-06-03T06:51:51.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:51:52.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836550 2024-06-03T06:51:52.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:51:52.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673022 2024-06-03T06:51:52.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:51:52.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:51:52.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673022 2024-06-03T06:51:52.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:51:52.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673022 2024-06-03T06:51:52.213 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673022 2024-06-03T06:51:52.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673022' 2024-06-03T06:51:52.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:51:52.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673021 -lt 42949673022 2024-06-03T06:51:52.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:51:53.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:51:53.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:51:53.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673023 -lt 42949673022 2024-06-03T06:51:53.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:51:53.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509496 2024-06-03T06:51:53.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:51:53.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:51:53.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509496 2024-06-03T06:51:53.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:51:53.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509496 2024-06-03T06:51:53.807 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509496 2024-06-03T06:51:53.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2024-06-03T06:51:53.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:51:54.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509497 -lt 64424509496 2024-06-03T06:51:54.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:51:54.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345967 2024-06-03T06:51:54.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:51:54.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:51:54.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345967 2024-06-03T06:51:54.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:51:54.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345967 2024-06-03T06:51:54.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345967' 2024-06-03T06:51:54.123 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345967 2024-06-03T06:51:54.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:51:54.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345967 -lt 85899345967 2024-06-03T06:51:54.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:51:54.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182440 2024-06-03T06:51:54.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:51:54.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:51:54.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182440 2024-06-03T06:51:54.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:51:54.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182440 2024-06-03T06:51:54.420 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182440 2024-06-03T06:51:54.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182440' 2024-06-03T06:51:54.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:51:54.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182440 -lt 107374182440 2024-06-03T06:51:54.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:51:54.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018913 2024-06-03T06:51:54.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:51:54.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:51:54.713 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018913 2024-06-03T06:51:54.713 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:51:54.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018913 2024-06-03T06:51:54.714 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018913 2024-06-03T06:51:54.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018913' 2024-06-03T06:51:54.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:51:55.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018913 -lt 128849018913 2024-06-03T06:51:55.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:51:55.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855389 2024-06-03T06:51:55.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:51:55.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:51:55.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855389 2024-06-03T06:51:55.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:51:55.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855389 2024-06-03T06:51:55.007 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.6 seq 150323855389 2024-06-03T06:51:55.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855389' 2024-06-03T06:51:55.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:51:55.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855390 -lt 150323855389 2024-06-03T06:51:55.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:51:55.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691860 2024-06-03T06:51:55.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:51:55.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:51:55.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691860 2024-06-03T06:51:55.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:51:55.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691860 2024-06-03T06:51:55.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691860' 2024-06-03T06:51:55.321 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.7 seq 171798691860 2024-06-03T06:51:55.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:51:55.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691860 -lt 171798691860 2024-06-03T06:51:55.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:51:55.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528339 2024-06-03T06:51:55.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:51:55.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:51:55.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528339 2024-06-03T06:51:55.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:51:55.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528339 2024-06-03T06:51:55.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528339' 2024-06-03T06:51:55.616 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.8 seq 193273528339 2024-06-03T06:51:55.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:51:55.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528339 -lt 193273528339 2024-06-03T06:51:55.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:51:55.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397510 2024-06-03T06:51:55.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:51:55.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:51:55.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397510 2024-06-03T06:51:55.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:51:55.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397510 2024-06-03T06:51:55.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397510' 2024-06-03T06:51:55.910 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.9 seq 210453397510 2024-06-03T06:51:55.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:51:56.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397510 -lt 210453397510 2024-06-03T06:51:56.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:51:56.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:51:56.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:51:56.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T06:51:56.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:51:56.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:51:56.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:51:56.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:51:56.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:51:56.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:51:56.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:51:56.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:51:56.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:51:56.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:51:56.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:51:57.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T06:51:57.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:51:57.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:51:57.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-06-03T06:51:57.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-03T06:51:57.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-03T06:51:57.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-06-03T06:51:57.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-03T06:51:57.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-06-03T06:51:57.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-03T06:51:57.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-03T06:51:58.643 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'ecpool' created 2024-06-03T06:51:58.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:51:59.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-06-03T06:51:59.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-06-03T06:52:00.305 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 2 allow_ec_overwrites to true 2024-06-03T06:52:00.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-03T06:52:00.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:52:00.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:52:00.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:52:00.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:52:00.318 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:52:00.318 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:52:00.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:52:00.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:52:00.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:52:00.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:52:00.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:52:00.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:52:00.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:52:00.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:52:00.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:52:00.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:52:00.704 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:52:00.705 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:52:00.705 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:52:00.705 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-06-03T06:52:00.705 INFO:tasks.workunit.client.0.smithi039.stderr:5 2024-06-03T06:52:00.705 INFO:tasks.workunit.client.0.smithi039.stderr:6 2024-06-03T06:52:00.705 INFO:tasks.workunit.client.0.smithi039.stderr:7 2024-06-03T06:52:00.705 INFO:tasks.workunit.client.0.smithi039.stderr:8 2024-06-03T06:52:00.705 INFO:tasks.workunit.client.0.smithi039.stderr:9' 2024-06-03T06:52:00.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:52:00.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:00.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:52:00.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836553 2024-06-03T06:52:00.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836553 2024-06-03T06:52:00.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553' 2024-06-03T06:52:00.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:00.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:52:01.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673025 2024-06-03T06:52:01.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673025 2024-06-03T06:52:01.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025' 2024-06-03T06:52:01.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:01.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:52:01.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509499 2024-06-03T06:52:01.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509499 2024-06-03T06:52:01.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509499' 2024-06-03T06:52:01.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:01.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:52:01.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345970 2024-06-03T06:52:01.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345970 2024-06-03T06:52:01.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509499 3-85899345970' 2024-06-03T06:52:01.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:01.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:52:01.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182443 2024-06-03T06:52:01.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182443 2024-06-03T06:52:01.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509499 3-85899345970 4-107374182443' 2024-06-03T06:52:01.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:01.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:52:01.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018916 2024-06-03T06:52:01.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018916 2024-06-03T06:52:01.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509499 3-85899345970 4-107374182443 5-128849018916' 2024-06-03T06:52:01.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:01.777 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:52:01.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855393 2024-06-03T06:52:01.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855393 2024-06-03T06:52:01.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509499 3-85899345970 4-107374182443 5-128849018916 6-150323855393' 2024-06-03T06:52:01.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:01.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:52:02.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691863 2024-06-03T06:52:02.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691863 2024-06-03T06:52:02.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509499 3-85899345970 4-107374182443 5-128849018916 6-150323855393 7-171798691863' 2024-06-03T06:52:02.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:02.045 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:52:02.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528342 2024-06-03T06:52:02.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528342 2024-06-03T06:52:02.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509499 3-85899345970 4-107374182443 5-128849018916 6-150323855393 7-171798691863 8-193273528342' 2024-06-03T06:52:02.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:02.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:52:02.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397513 2024-06-03T06:52:02.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397513 2024-06-03T06:52:02.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509499 3-85899345970 4-107374182443 5-128849018916 6-150323855393 7-171798691863 8-193273528342 9-210453397513' 2024-06-03T06:52:02.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:02.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836553 2024-06-03T06:52:02.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:02.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:52:02.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836553 2024-06-03T06:52:02.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:02.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836553 2024-06-03T06:52:02.264 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836553 2024-06-03T06:52:02.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-06-03T06:52:02.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:52:02.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836553 -lt 21474836553 2024-06-03T06:52:02.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:02.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673025 2024-06-03T06:52:02.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:02.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:52:02.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673025 2024-06-03T06:52:02.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:02.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673025 2024-06-03T06:52:02.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673025' 2024-06-03T06:52:02.557 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673025 2024-06-03T06:52:02.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:52:02.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673024 -lt 42949673025 2024-06-03T06:52:02.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:52:03.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:52:03.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:52:04.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673026 -lt 42949673025 2024-06-03T06:52:04.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:04.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509499 2024-06-03T06:52:04.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:04.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:52:04.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509499 2024-06-03T06:52:04.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:04.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509499 2024-06-03T06:52:04.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509499' 2024-06-03T06:52:04.155 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509499 2024-06-03T06:52:04.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:52:04.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509500 -lt 64424509499 2024-06-03T06:52:04.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:04.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345970 2024-06-03T06:52:04.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:04.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:52:04.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345970 2024-06-03T06:52:04.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:04.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345970 2024-06-03T06:52:04.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345970' 2024-06-03T06:52:04.499 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345970 2024-06-03T06:52:04.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:52:04.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345970 -lt 85899345970 2024-06-03T06:52:04.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:04.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182443 2024-06-03T06:52:04.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:04.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:52:04.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182443 2024-06-03T06:52:04.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:04.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182443 2024-06-03T06:52:04.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182443' 2024-06-03T06:52:04.797 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182443 2024-06-03T06:52:04.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:52:05.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182443 -lt 107374182443 2024-06-03T06:52:05.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:05.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018916 2024-06-03T06:52:05.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:05.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:52:05.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018916 2024-06-03T06:52:05.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:05.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018916 2024-06-03T06:52:05.092 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018916 2024-06-03T06:52:05.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018916' 2024-06-03T06:52:05.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:52:05.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018916 -lt 128849018916 2024-06-03T06:52:05.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:05.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855393 2024-06-03T06:52:05.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:05.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:52:05.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855393 2024-06-03T06:52:05.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:05.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855393 2024-06-03T06:52:05.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855393' 2024-06-03T06:52:05.397 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.6 seq 150323855393 2024-06-03T06:52:05.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:52:05.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855393 -lt 150323855393 2024-06-03T06:52:05.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:05.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691863 2024-06-03T06:52:05.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:05.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:52:05.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691863 2024-06-03T06:52:05.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:05.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691863 2024-06-03T06:52:05.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691863' 2024-06-03T06:52:05.692 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.7 seq 171798691863 2024-06-03T06:52:05.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:52:06.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691863 -lt 171798691863 2024-06-03T06:52:06.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:06.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528342 2024-06-03T06:52:06.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:06.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:52:06.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528342 2024-06-03T06:52:06.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:06.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528342 2024-06-03T06:52:06.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528342' 2024-06-03T06:52:06.049 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.8 seq 193273528342 2024-06-03T06:52:06.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:52:06.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528342 -lt 193273528342 2024-06-03T06:52:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:06.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397513 2024-06-03T06:52:06.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:52:06.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397513 2024-06-03T06:52:06.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:06.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397513 2024-06-03T06:52:06.352 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.9 seq 210453397513 2024-06-03T06:52:06.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397513' 2024-06-03T06:52:06.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:52:06.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397513 -lt 210453397513 2024-06-03T06:52:06.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:52:06.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:52:06.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:52:07.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:52:07.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:52:07.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:52:07.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:52:07.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:52:07.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:52:07.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:52:07.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:52:07.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:52:07.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:52:07.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:52:07.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:52:07.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:52:07.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:52:07.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:52:07.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-03T06:52:07.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-06-03T06:52:07.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-03T06:52:07.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-06-03T06:52:07.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-06-03T06:52:07.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T06:52:07.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T06:52:07.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-03T06:52:07.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T06:52:07.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T06:52:07.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T06:52:08.093 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T06:52:08.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T06:52:08.629 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T06:52:08.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T06:52:08.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T06:52:08.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-03T06:52:09.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-06-03T06:52:09.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-03T06:52:09.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:52:09.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:52:09.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:52:09.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-06-03T06:52:09.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-06-03T06:52:09.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-03T06:52:09.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-06-03T06:52:09.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-06-03T06:52:09.449 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:52:09.450 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T06:52:09.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-06-03T06:52:09.740 INFO:tasks.workunit.client.0.smithi039.stderr:5 2024-06-03T06:52:09.740 INFO:tasks.workunit.client.0.smithi039.stderr:9 2024-06-03T06:52:09.740 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:52:09.740 INFO:tasks.workunit.client.0.smithi039.stderr:6 2024-06-03T06:52:09.741 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:52:09.741 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:52:09.741 INFO:tasks.workunit.client.0.smithi039.stderr:7' 2024-06-03T06:52:09.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-06-03T06:52:09.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-06-03T06:52:09.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-06-03T06:52:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-06-03T06:52:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-06-03T06:52:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-06-03T06:52:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-03T06:52:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-03T06:52:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-06-03T06:52:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:52:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:52:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:52:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:52:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:52:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:52:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:52:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:52:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:52:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:52:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:52:09.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:52:09.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:52:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:52:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:52:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:52:10.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:52:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:52:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:52:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:52:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:52:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:52:10.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:52:10.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-03T06:52:12.248 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-06-03T06:52:12.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:52:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:52:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:52:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:52:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:52:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:52:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:52:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:52:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:52:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:52:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:52:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:52:12.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:52:12.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:52:12.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:52:12.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:52:12.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:52:12.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:52:12.775 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:52:12.775 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:52:12.775 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:52:12.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:52:12.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:52:12.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:52:12.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:52:12.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:52:12.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:52:12.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:52:12.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:52:12.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:52:12.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:52:12.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:52:12.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:52:12.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:52:12.778 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.3 2024-06-03T06:52:12.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:52:12.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:52:12.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:52:12.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:52:12.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:52:12.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:52:12.807 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:52:12.807+0000 7efca3918640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:52:12.810 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:52:12.810+0000 7efca3918640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:52:12.811 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:52:12.811+0000 7efca3918640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:52:13.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:52:13.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:52:13.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:52:13.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:52:13.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:52:13.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:52:13.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:52:13.083 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:52:13.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:52:13.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:52:13.327 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:52:13.327+0000 7efca3918640 -1 Falling back to public interface 2024-06-03T06:52:13.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:52:14.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:52:14.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:52:14.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:52:14.373 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:52:14.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:52:14.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:52:14.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:52:15.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:52:15.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:52:15.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:52:15.674 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:52:15.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:52:15.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:52:15.764 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:52:15.764+0000 7efca3918640 -1 osd.3 67 log_to_monitors true 2024-06-03T06:52:15.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:52:16.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:52:16.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:52:16.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:52:16.996 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:52:16.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:52:16.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:52:17.475 INFO:tasks.workunit.client.0.smithi039.stderr:osd.3 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [20,67) [v2:127.0.0.1:6826/602028070,v1:127.0.0.1:6827/602028070] [v2:127.0.0.1:6828/602028070,v1:127.0.0.1:6829/602028070] exists,up 79666bc5-58ed-418d-906d-594dd41a932e 2024-06-03T06:52:17.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:52:17.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:52:17.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:52:17.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:52:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:52:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:52:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:52:17.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:52:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:52:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:52:17.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:52:17.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:52:17.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:52:17.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:52:17.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:52:17.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:52:17.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:52:17.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:52:17.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:52:17.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:52:17.910 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:52:17.910 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:52:17.910 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:52:17.910 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-06-03T06:52:17.911 INFO:tasks.workunit.client.0.smithi039.stderr:5 2024-06-03T06:52:17.911 INFO:tasks.workunit.client.0.smithi039.stderr:6 2024-06-03T06:52:17.911 INFO:tasks.workunit.client.0.smithi039.stderr:7 2024-06-03T06:52:17.911 INFO:tasks.workunit.client.0.smithi039.stderr:8 2024-06-03T06:52:17.911 INFO:tasks.workunit.client.0.smithi039.stderr:9' 2024-06-03T06:52:17.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:52:17.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:17.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:52:18.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836557 2024-06-03T06:52:18.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836557 2024-06-03T06:52:18.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557' 2024-06-03T06:52:18.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:18.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:52:18.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673030 2024-06-03T06:52:18.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673030 2024-06-03T06:52:18.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673030' 2024-06-03T06:52:18.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:18.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:52:18.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509504 2024-06-03T06:52:18.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509504 2024-06-03T06:52:18.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673030 2-64424509504' 2024-06-03T06:52:18.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:18.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:52:18.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678018 2024-06-03T06:52:18.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678018 2024-06-03T06:52:18.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673030 2-64424509504 3-304942678018' 2024-06-03T06:52:18.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:18.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:52:18.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182447 2024-06-03T06:52:18.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182447 2024-06-03T06:52:18.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673030 2-64424509504 3-304942678018 4-107374182447' 2024-06-03T06:52:18.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:18.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:52:18.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018920 2024-06-03T06:52:18.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018920 2024-06-03T06:52:18.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673030 2-64424509504 3-304942678018 4-107374182447 5-128849018920' 2024-06-03T06:52:18.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:18.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:52:18.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855397 2024-06-03T06:52:18.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855397 2024-06-03T06:52:18.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673030 2-64424509504 3-304942678018 4-107374182447 5-128849018920 6-150323855397' 2024-06-03T06:52:18.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:18.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:52:18.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691867 2024-06-03T06:52:18.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691867 2024-06-03T06:52:18.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673030 2-64424509504 3-304942678018 4-107374182447 5-128849018920 6-150323855397 7-171798691867' 2024-06-03T06:52:18.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:18.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:52:18.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528346 2024-06-03T06:52:18.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528346 2024-06-03T06:52:18.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673030 2-64424509504 3-304942678018 4-107374182447 5-128849018920 6-150323855397 7-171798691867 8-193273528346' 2024-06-03T06:52:18.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:18.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:52:19.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397517 2024-06-03T06:52:19.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397517 2024-06-03T06:52:19.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673030 2-64424509504 3-304942678018 4-107374182447 5-128849018920 6-150323855397 7-171798691867 8-193273528346 9-210453397517' 2024-06-03T06:52:19.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:19.035 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836557 2024-06-03T06:52:19.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:19.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:52:19.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836557 2024-06-03T06:52:19.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:19.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836557 2024-06-03T06:52:19.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836557' 2024-06-03T06:52:19.038 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836557 2024-06-03T06:52:19.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:52:19.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836558 -lt 21474836557 2024-06-03T06:52:19.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:19.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673030 2024-06-03T06:52:19.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:19.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:52:19.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673030 2024-06-03T06:52:19.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:19.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673030 2024-06-03T06:52:19.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673030' 2024-06-03T06:52:19.353 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949673030 2024-06-03T06:52:19.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:52:19.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673030 -lt 42949673030 2024-06-03T06:52:19.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:19.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509504 2024-06-03T06:52:19.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:19.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:52:19.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509504 2024-06-03T06:52:19.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:19.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509504 2024-06-03T06:52:19.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509504' 2024-06-03T06:52:19.645 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509504 2024-06-03T06:52:19.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:52:19.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509504 -lt 64424509504 2024-06-03T06:52:19.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:19.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678018 2024-06-03T06:52:19.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:19.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:52:19.943 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678018 2024-06-03T06:52:19.943 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:19.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678018 2024-06-03T06:52:19.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678018' 2024-06-03T06:52:19.945 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 304942678018 2024-06-03T06:52:19.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:52:20.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678018 -lt 304942678018 2024-06-03T06:52:20.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:20.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182447 2024-06-03T06:52:20.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:20.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:52:20.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182447 2024-06-03T06:52:20.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:20.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182447 2024-06-03T06:52:20.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182447' 2024-06-03T06:52:20.276 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.4 seq 107374182447 2024-06-03T06:52:20.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:52:20.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182447 -lt 107374182447 2024-06-03T06:52:20.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:20.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018920 2024-06-03T06:52:20.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:20.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:52:20.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018920 2024-06-03T06:52:20.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:20.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018920 2024-06-03T06:52:20.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018920' 2024-06-03T06:52:20.569 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.5 seq 128849018920 2024-06-03T06:52:20.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:52:20.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018920 -lt 128849018920 2024-06-03T06:52:20.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:20.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855397 2024-06-03T06:52:20.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:20.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:52:20.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855397 2024-06-03T06:52:20.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:20.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855397 2024-06-03T06:52:20.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855397' 2024-06-03T06:52:20.861 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.6 seq 150323855397 2024-06-03T06:52:20.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:52:21.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855397 -lt 150323855397 2024-06-03T06:52:21.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:21.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691867 2024-06-03T06:52:21.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:21.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:52:21.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691867 2024-06-03T06:52:21.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:21.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691867 2024-06-03T06:52:21.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691867' 2024-06-03T06:52:21.156 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.7 seq 171798691867 2024-06-03T06:52:21.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:52:21.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691867 -lt 171798691867 2024-06-03T06:52:21.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:21.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528346 2024-06-03T06:52:21.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:21.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:52:21.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528346 2024-06-03T06:52:21.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:21.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528346 2024-06-03T06:52:21.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528346' 2024-06-03T06:52:21.453 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.8 seq 193273528346 2024-06-03T06:52:21.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:52:21.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528347 -lt 193273528346 2024-06-03T06:52:21.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:21.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397517 2024-06-03T06:52:21.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:21.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:52:21.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397517 2024-06-03T06:52:21.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:21.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397517 2024-06-03T06:52:21.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397517' 2024-06-03T06:52:21.744 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.9 seq 210453397517 2024-06-03T06:52:21.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:52:22.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397518 -lt 210453397517 2024-06-03T06:52:22.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:52:22.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:52:22.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:52:22.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:52:22.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:52:22.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:52:22.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:52:22.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:52:22.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:52:22.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:52:22.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:52:22.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:52:22.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:52:22.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:52:22.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:52:23.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:52:23.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:52:23.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:52:23.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-03T06:52:23.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:52:23.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:52:23.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:52:23.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:52:23.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-03T06:52:23.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-03T06:52:23.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:52:23.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:52:23.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:52:23.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:52:23.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:52:23.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:52:23.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:51:58.383451+0000 2024-06-03T06:52:23.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:52:23.978 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:52:23.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:51:58.383451+0000 2024-06-03T06:52:23.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:52:23.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:51:58.383451+0000 2024-06-03T06:52:23.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:52:23.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:52:23.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:52:23.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:52:23.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:52:23.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:52:23.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:52:23.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:52:24.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:51:58.383451+0000 '>' 2024-06-03T06:51:58.383451+0000 2024-06-03T06:52:24.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:52:25.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:52:25.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:52:25.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:52:25.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:52:25.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:52:25.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:52:25.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:52:25.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:51:58.383451+0000 '>' 2024-06-03T06:51:58.383451+0000 2024-06-03T06:52:25.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:52:26.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:52:26.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:52:26.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:52:26.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:52:26.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:52:26.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:52:26.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:52:26.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:51:58.383451+0000 '>' 2024-06-03T06:51:58.383451+0000 2024-06-03T06:52:26.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:52:27.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:52:27.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:52:27.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:52:27.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:52:27.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:52:27.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:52:27.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:52:28.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:52:24.705712+0000 '>' 2024-06-03T06:51:58.383451+0000 2024-06-03T06:52:28.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:52:28.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:52:28.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:52:28.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:52:28.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-03T06:52:28.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:52:28.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:52:28.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:52:28.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:52:28.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-03T06:52:28.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:52:28.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-03T06:52:28.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:52:28.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:52:28.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:52:28.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:52:28.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:52:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:52:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:52:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:52:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:52:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-03T06:52:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:52:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:52:28.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-03T06:52:30.540 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:52:30.540 INFO:tasks.workunit.client.0.smithi039.stdout:hinfo_key 2024-06-03T06:52:30.540 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:52:30.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-03T06:52:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:52:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:52:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-03T06:52:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:52:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:52:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:52:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:52:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:52:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:52:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-03T06:52:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-03T06:52:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:52:30.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:52:30.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:52:30.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:52:30.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:52:30.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:52:30.814 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:52:30.814 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:52:30.814 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:52:30.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:52:30.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:52:30.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:52:30.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:52:30.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:52:30.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:52:30.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:52:30.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:52:30.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:52:30.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:52:30.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:52:30.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-03T06:52:30.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-03T06:52:30.816 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.9 2024-06-03T06:52:30.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:52:30.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-03T06:52:30.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-03T06:52:30.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:52:30.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:52:30.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:52:30.845 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:52:30.845+0000 7f763f20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:52:30.848 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:52:30.848+0000 7f763f20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:52:30.849 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:52:30.849+0000 7f763f20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:52:31.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-03T06:52:31.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:52:31.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-03T06:52:31.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:52:31.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:52:31.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:52:31.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:52:31.122 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:52:31.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:52:31.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:52:31.368 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:52:31.368+0000 7f763f20d640 -1 Falling back to public interface 2024-06-03T06:52:31.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:52:32.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:52:32.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:52:32.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:52:32.415 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:52:32.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:52:32.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:52:32.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:52:33.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:52:33.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:52:33.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:52:33.713 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:52:33.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:52:33.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:52:33.720 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:52:33.720+0000 7f763f20d640 -1 osd.9 72 log_to_monitors true 2024-06-03T06:52:34.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:52:35.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:52:35.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:52:35.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:52:35.032 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:52:35.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:52:35.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:52:35.335 INFO:tasks.workunit.client.0.smithi039.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 73 last_clean_interval [49,72) [v2:127.0.0.1:6874/3101823092,v1:127.0.0.1:6875/3101823092] [v2:127.0.0.1:6876/3101823092,v1:127.0.0.1:6877/3101823092] exists,up fc630115-1953-4e3d-885f-d2824414ef72 2024-06-03T06:52:35.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:52:35.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:52:35.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:52:35.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:52:35.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:52:35.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:52:35.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:52:35.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:52:35.336 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:52:35.336 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:52:35.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:52:35.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:52:35.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:52:35.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:52:35.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:52:35.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:52:35.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:52:35.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:52:35.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:52:36.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:52:36.021 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:52:36.022 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:52:36.022 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:52:36.022 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-06-03T06:52:36.022 INFO:tasks.workunit.client.0.smithi039.stderr:5 2024-06-03T06:52:36.022 INFO:tasks.workunit.client.0.smithi039.stderr:6 2024-06-03T06:52:36.022 INFO:tasks.workunit.client.0.smithi039.stderr:7 2024-06-03T06:52:36.022 INFO:tasks.workunit.client.0.smithi039.stderr:8 2024-06-03T06:52:36.022 INFO:tasks.workunit.client.0.smithi039.stderr:9' 2024-06-03T06:52:36.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:52:36.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:36.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:52:36.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836562 2024-06-03T06:52:36.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836562 2024-06-03T06:52:36.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562' 2024-06-03T06:52:36.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:36.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:52:36.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673034 2024-06-03T06:52:36.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673034 2024-06-03T06:52:36.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673034' 2024-06-03T06:52:36.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:36.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:52:36.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509508 2024-06-03T06:52:36.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509508 2024-06-03T06:52:36.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673034 2-64424509508' 2024-06-03T06:52:36.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:36.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:52:36.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678023 2024-06-03T06:52:36.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678023 2024-06-03T06:52:36.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673034 2-64424509508 3-304942678023' 2024-06-03T06:52:36.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:36.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:52:36.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182452 2024-06-03T06:52:36.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182452 2024-06-03T06:52:36.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673034 2-64424509508 3-304942678023 4-107374182452' 2024-06-03T06:52:36.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:36.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:52:36.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018925 2024-06-03T06:52:36.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018925 2024-06-03T06:52:36.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673034 2-64424509508 3-304942678023 4-107374182452 5-128849018925' 2024-06-03T06:52:36.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:36.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:52:36.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855402 2024-06-03T06:52:36.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855402 2024-06-03T06:52:36.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673034 2-64424509508 3-304942678023 4-107374182452 5-128849018925 6-150323855402' 2024-06-03T06:52:36.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:36.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:52:37.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691872 2024-06-03T06:52:37.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691872 2024-06-03T06:52:37.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673034 2-64424509508 3-304942678023 4-107374182452 5-128849018925 6-150323855402 7-171798691872' 2024-06-03T06:52:37.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:37.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:52:37.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528351 2024-06-03T06:52:37.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528351 2024-06-03T06:52:37.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673034 2-64424509508 3-304942678023 4-107374182452 5-128849018925 6-150323855402 7-171798691872 8-193273528351' 2024-06-03T06:52:37.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:37.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:52:37.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-06-03T06:52:37.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-06-03T06:52:37.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673034 2-64424509508 3-304942678023 4-107374182452 5-128849018925 6-150323855402 7-171798691872 8-193273528351 9-330712481794' 2024-06-03T06:52:37.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:37.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836562 2024-06-03T06:52:37.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:37.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:52:37.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836562 2024-06-03T06:52:37.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:37.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836562 2024-06-03T06:52:37.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836562' 2024-06-03T06:52:37.356 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836562 2024-06-03T06:52:37.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:52:37.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836562 -lt 21474836562 2024-06-03T06:52:37.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:37.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673034 2024-06-03T06:52:37.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:37.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:52:37.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673034 2024-06-03T06:52:37.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:37.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673034 2024-06-03T06:52:37.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673034' 2024-06-03T06:52:37.648 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949673034 2024-06-03T06:52:37.648 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:52:37.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673034 -lt 42949673034 2024-06-03T06:52:37.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:37.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509508 2024-06-03T06:52:37.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:37.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:52:37.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509508 2024-06-03T06:52:37.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:37.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509508 2024-06-03T06:52:37.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509508' 2024-06-03T06:52:37.939 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509508 2024-06-03T06:52:37.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:52:38.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509508 -lt 64424509508 2024-06-03T06:52:38.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:38.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678023 2024-06-03T06:52:38.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:38.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:52:38.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678023 2024-06-03T06:52:38.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:38.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678023 2024-06-03T06:52:38.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678023' 2024-06-03T06:52:38.270 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 304942678023 2024-06-03T06:52:38.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:52:38.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678023 -lt 304942678023 2024-06-03T06:52:38.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:38.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182452 2024-06-03T06:52:38.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:38.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:52:38.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182452 2024-06-03T06:52:38.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:38.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182452 2024-06-03T06:52:38.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182452' 2024-06-03T06:52:38.564 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.4 seq 107374182452 2024-06-03T06:52:38.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:52:38.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182452 -lt 107374182452 2024-06-03T06:52:38.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:38.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018925 2024-06-03T06:52:38.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:38.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:52:38.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018925 2024-06-03T06:52:38.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:38.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018925 2024-06-03T06:52:38.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018925' 2024-06-03T06:52:38.859 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.5 seq 128849018925 2024-06-03T06:52:38.859 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:52:39.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018925 -lt 128849018925 2024-06-03T06:52:39.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:39.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855402 2024-06-03T06:52:39.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:39.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:52:39.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855402 2024-06-03T06:52:39.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:39.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855402 2024-06-03T06:52:39.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855402' 2024-06-03T06:52:39.154 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.6 seq 150323855402 2024-06-03T06:52:39.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:52:39.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855402 -lt 150323855402 2024-06-03T06:52:39.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:39.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691872 2024-06-03T06:52:39.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:39.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:52:39.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691872 2024-06-03T06:52:39.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:39.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691872 2024-06-03T06:52:39.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691872' 2024-06-03T06:52:39.450 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.7 seq 171798691872 2024-06-03T06:52:39.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:52:39.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691872 -lt 171798691872 2024-06-03T06:52:39.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:39.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528351 2024-06-03T06:52:39.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:39.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:52:39.743 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528351 2024-06-03T06:52:39.743 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:39.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528351 2024-06-03T06:52:39.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528351' 2024-06-03T06:52:39.744 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.8 seq 193273528351 2024-06-03T06:52:39.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:52:40.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528351 -lt 193273528351 2024-06-03T06:52:40.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:40.045 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481794 2024-06-03T06:52:40.045 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:40.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:52:40.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481794 2024-06-03T06:52:40.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:40.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-06-03T06:52:40.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-06-03T06:52:40.048 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.9 seq 330712481794 2024-06-03T06:52:40.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:52:40.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481794 -lt 330712481794 2024-06-03T06:52:40.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:52:40.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:52:40.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:52:40.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:52:40.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:52:40.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:52:40.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:52:40.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:52:40.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:52:40.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:52:40.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:52:40.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:52:40.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:52:40.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:52:40.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:52:41.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:52:41.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:52:41.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:52:41.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:52:41.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:52:41.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-06-03T06:52:41.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-03T06:52:41.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-03T06:52:41.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-06-03T06:52:41.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:52:41.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:52:41.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:52:41.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-03T06:52:41.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:52:41.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:52:41.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:52:41.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:52:41.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-03T06:52:41.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:52:41.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-03T06:52:41.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:52:41.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:52:41.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:52:41.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:52:41.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:52:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:52:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:52:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:52:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:52:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-03T06:52:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:52:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:52:41.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-03T06:52:44.357 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-03T06:52:44.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-03T06:52:44.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:52:44.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:52:44.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-03T06:52:44.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:52:44.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:52:44.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:52:44.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:52:44.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:52:44.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:52:44.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-03T06:52:44.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-03T06:52:44.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:52:44.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:52:44.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:52:44.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:52:44.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:52:44.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:52:44.891 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:52:44.891 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:52:44.891 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:52:44.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:52:44.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:52:44.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:52:44.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:52:44.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:52:44.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:52:44.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:52:44.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:52:44.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:52:44.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:52:44.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:52:44.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-03T06:52:44.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-03T06:52:44.894 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.5 2024-06-03T06:52:44.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:52:44.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-03T06:52:44.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-03T06:52:44.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:52:44.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:52:44.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:52:44.915 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:52:44.914+0000 7fc162a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:52:44.917 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:52:44.916+0000 7fc162a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:52:44.919 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:52:44.918+0000 7fc162a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:52:45.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-03T06:52:45.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:52:45.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-03T06:52:45.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:52:45.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:52:45.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:52:45.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:52:45.191 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:52:45.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:52:45.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:52:45.435 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:52:45.434+0000 7fc162a0d640 -1 Falling back to public interface 2024-06-03T06:52:45.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:52:46.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:52:46.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:52:46.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:52:46.505 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:52:46.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:52:46.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:52:46.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:52:47.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:52:47.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:52:47.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:52:47.804 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:52:47.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:52:47.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:52:47.892 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:52:47.891+0000 7fc162a0d640 -1 osd.5 79 log_to_monitors true 2024-06-03T06:52:48.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:52:49.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:52:49.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:52:49.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:52:49.106 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:52:49.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:52:49.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:52:49.608 INFO:tasks.workunit.client.0.smithi039.stderr:osd.5 up in weight 1 up_from 84 up_thru 84 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/1824161250,v1:127.0.0.1:6843/1824161250] [v2:127.0.0.1:6844/1824161250,v1:127.0.0.1:6845/1824161250] exists,up 915fce46-b35c-47e4-a164-8e162d251809 2024-06-03T06:52:49.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:52:49.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:52:49.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:52:49.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:52:49.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:52:49.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:52:49.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:52:49.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:52:49.609 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:52:49.609 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:52:49.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:52:49.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:52:49.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:52:49.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:52:49.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:52:49.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:52:49.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:52:49.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:52:49.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:52:50.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:52:50.152 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:52:50.152 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:52:50.152 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:52:50.152 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-06-03T06:52:50.152 INFO:tasks.workunit.client.0.smithi039.stderr:5 2024-06-03T06:52:50.152 INFO:tasks.workunit.client.0.smithi039.stderr:6 2024-06-03T06:52:50.152 INFO:tasks.workunit.client.0.smithi039.stderr:7 2024-06-03T06:52:50.152 INFO:tasks.workunit.client.0.smithi039.stderr:8 2024-06-03T06:52:50.152 INFO:tasks.workunit.client.0.smithi039.stderr:9' 2024-06-03T06:52:50.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:52:50.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:50.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:52:50.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836566 2024-06-03T06:52:50.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836566 2024-06-03T06:52:50.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566' 2024-06-03T06:52:50.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:50.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:52:50.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673038 2024-06-03T06:52:50.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673038 2024-06-03T06:52:50.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673038' 2024-06-03T06:52:50.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:50.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:52:50.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509512 2024-06-03T06:52:50.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509512 2024-06-03T06:52:50.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509512' 2024-06-03T06:52:50.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:50.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:52:50.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678026 2024-06-03T06:52:50.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678026 2024-06-03T06:52:50.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509512 3-304942678026' 2024-06-03T06:52:50.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:50.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:52:50.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182455 2024-06-03T06:52:50.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182455 2024-06-03T06:52:50.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509512 3-304942678026 4-107374182455' 2024-06-03T06:52:50.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:50.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:52:50.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-06-03T06:52:50.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-06-03T06:52:50.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509512 3-304942678026 4-107374182455 5-360777252866' 2024-06-03T06:52:50.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:50.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:52:51.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855405 2024-06-03T06:52:51.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855405 2024-06-03T06:52:51.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509512 3-304942678026 4-107374182455 5-360777252866 6-150323855405' 2024-06-03T06:52:51.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:51.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:52:51.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691876 2024-06-03T06:52:51.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691876 2024-06-03T06:52:51.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509512 3-304942678026 4-107374182455 5-360777252866 6-150323855405 7-171798691876' 2024-06-03T06:52:51.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:51.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:52:51.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528355 2024-06-03T06:52:51.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528355 2024-06-03T06:52:51.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509512 3-304942678026 4-107374182455 5-360777252866 6-150323855405 7-171798691876 8-193273528355' 2024-06-03T06:52:51.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:52:51.307 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:52:51.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481798 2024-06-03T06:52:51.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481798 2024-06-03T06:52:51.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509512 3-304942678026 4-107374182455 5-360777252866 6-150323855405 7-171798691876 8-193273528355 9-330712481798' 2024-06-03T06:52:51.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:51.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836566 2024-06-03T06:52:51.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:51.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:52:51.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836566 2024-06-03T06:52:51.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:51.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836566 2024-06-03T06:52:51.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836566' 2024-06-03T06:52:51.419 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836566 2024-06-03T06:52:51.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:52:51.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836566 -lt 21474836566 2024-06-03T06:52:51.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:51.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673038 2024-06-03T06:52:51.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:51.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:52:51.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673038 2024-06-03T06:52:51.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:51.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673038 2024-06-03T06:52:51.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-06-03T06:52:51.711 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949673038 2024-06-03T06:52:51.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:52:52.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673038 -lt 42949673038 2024-06-03T06:52:52.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:52.068 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509512 2024-06-03T06:52:52.068 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:52.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:52:52.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509512 2024-06-03T06:52:52.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:52.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509512 2024-06-03T06:52:52.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2024-06-03T06:52:52.070 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509512 2024-06-03T06:52:52.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:52:52.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509512 -lt 64424509512 2024-06-03T06:52:52.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:52.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678026 2024-06-03T06:52:52.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:52.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:52:52.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678026 2024-06-03T06:52:52.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:52.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678026 2024-06-03T06:52:52.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678026' 2024-06-03T06:52:52.374 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 304942678026 2024-06-03T06:52:52.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:52:52.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678026 -lt 304942678026 2024-06-03T06:52:52.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:52.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182455 2024-06-03T06:52:52.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:52.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:52:52.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182455 2024-06-03T06:52:52.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:52.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182455 2024-06-03T06:52:52.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182455' 2024-06-03T06:52:52.665 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.4 seq 107374182455 2024-06-03T06:52:52.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:52:52.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182455 -lt 107374182455 2024-06-03T06:52:52.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:52.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252866 2024-06-03T06:52:52.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:52.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:52:52.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252866 2024-06-03T06:52:52.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:52.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-06-03T06:52:52.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252866' 2024-06-03T06:52:52.957 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.5 seq 360777252866 2024-06-03T06:52:52.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:52:53.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252866 -lt 360777252866 2024-06-03T06:52:53.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:53.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855405 2024-06-03T06:52:53.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:53.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:52:53.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855405 2024-06-03T06:52:53.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:53.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855405 2024-06-03T06:52:53.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855405' 2024-06-03T06:52:53.249 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.6 seq 150323855405 2024-06-03T06:52:53.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:52:53.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855406 -lt 150323855405 2024-06-03T06:52:53.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:53.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691876 2024-06-03T06:52:53.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:53.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:52:53.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691876 2024-06-03T06:52:53.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:53.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691876 2024-06-03T06:52:53.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691876' 2024-06-03T06:52:53.580 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.7 seq 171798691876 2024-06-03T06:52:53.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:52:53.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691876 -lt 171798691876 2024-06-03T06:52:53.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:53.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528355 2024-06-03T06:52:53.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:53.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:52:53.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528355 2024-06-03T06:52:53.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:53.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528355 2024-06-03T06:52:53.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528355' 2024-06-03T06:52:53.875 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.8 seq 193273528355 2024-06-03T06:52:53.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:52:54.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528355 -lt 193273528355 2024-06-03T06:52:54.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:52:54.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481798 2024-06-03T06:52:54.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:52:54.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:52:54.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481798 2024-06-03T06:52:54.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:52:54.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481798 2024-06-03T06:52:54.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-06-03T06:52:54.211 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.9 seq 330712481798 2024-06-03T06:52:54.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:52:54.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481798 -lt 330712481798 2024-06-03T06:52:54.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:52:54.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:52:54.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:52:54.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:52:54.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:52:54.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:52:54.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:52:54.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:52:54.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:52:54.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:52:54.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:52:55.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:52:55.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:52:55.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:52:55.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:52:55.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:52:55.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:52:55.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:52:55.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-03T06:52:55.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:52:55.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:52:55.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:52:55.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:52:55.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-03T06:52:55.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-03T06:52:55.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:52:55.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:52:55.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:52:55.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:52:55.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:52:55.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:52:56.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:52:24.705712+0000 2024-06-03T06:52:56.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:52:56.381 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:52:56.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:52:24.705712+0000 2024-06-03T06:52:56.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:52:56.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:52:24.705712+0000 2024-06-03T06:52:56.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:52:56.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:52:56.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:52:56.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:52:56.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:52:56.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:52:56.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:52:56.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:52:56.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:52:24.705712+0000 '>' 2024-06-03T06:52:24.705712+0000 2024-06-03T06:52:56.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:52:57.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:52:57.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:52:57.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:52:57.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:52:57.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:52:57.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:52:57.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:52:57.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:52:24.705712+0000 '>' 2024-06-03T06:52:24.705712+0000 2024-06-03T06:52:57.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:52:58.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:52:58.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:52:58.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:52:58.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:52:58.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:52:58.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:52:58.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:52:59.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:52:24.705712+0000 '>' 2024-06-03T06:52:24.705712+0000 2024-06-03T06:52:59.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:53:00.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:53:00.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:53:00.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:53:00.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:53:00.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:53:00.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:53:00.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:53:00.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:52:24.705712+0000 '>' 2024-06-03T06:52:24.705712+0000 2024-06-03T06:53:00.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:53:01.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:53:01.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:53:01.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:53:01.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:53:01.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:53:01.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:53:01.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:53:01.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:52:24.705712+0000 '>' 2024-06-03T06:52:24.705712+0000 2024-06-03T06:53:01.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:53:02.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:53:02.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:53:02.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:53:02.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:53:02.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:53:02.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:53:02.880 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:53:03.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:52:56.787674+0000 '>' 2024-06-03T06:52:24.705712+0000 2024-06-03T06:53:03.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:53:03.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:53:03.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:53:03.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:53:03.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-03T06:53:03.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:53:03.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:53:03.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:53:03.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:53:03.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-03T06:53:03.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:53:03.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-03T06:53:03.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:53:03.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:53:03.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:53:03.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:53:03.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:53:03.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:53:03.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:53:03.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:53:03.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:53:03.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-03T06:53:03.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:53:03.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:53:03.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-03T06:53:05.429 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:53:05.430 INFO:tasks.workunit.client.0.smithi039.stdout:hinfo_key 2024-06-03T06:53:05.430 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:53:05.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-03T06:53:05.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:53:05.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:53:05.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-03T06:53:05.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:53:05.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:53:05.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:53:05.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:53:05.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:53:05.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:53:05.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-03T06:53:05.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-03T06:53:05.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:53:05.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:53:05.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:53:05.697 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:53:05.697 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:53:05.697 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:53:05.697 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:53:05.697 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:53:05.697 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:53:05.697 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:53:05.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:53:05.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:53:05.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:53:05.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:53:05.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:53:05.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:53:05.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:53:05.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:53:05.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:53:05.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:53:05.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-03T06:53:05.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-03T06:53:05.699 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.9 2024-06-03T06:53:05.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:53:05.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-03T06:53:05.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-03T06:53:05.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:53:05.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:53:05.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:53:05.729 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:53:05.729+0000 7f7a64a23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:53:05.731 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:53:05.732+0000 7f7a64a23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:53:05.733 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:53:05.733+0000 7f7a64a23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:53:06.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-03T06:53:06.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:53:06.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-03T06:53:06.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:53:06.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:53:06.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:06.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:53:06.001 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:53:06.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:06.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:53:06.249 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:53:06.249+0000 7f7a64a23640 -1 Falling back to public interface 2024-06-03T06:53:06.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:07.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:07.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:07.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:53:07.296 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:53:07.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:07.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:53:07.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:08.536 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:53:08.536+0000 7f7a64a23640 -1 osd.9 86 log_to_monitors true 2024-06-03T06:53:08.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:08.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:08.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:53:08.598 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:53:08.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:08.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:53:09.009 INFO:tasks.workunit.client.0.smithi039.stderr:osd.9 up in weight 1 up_from 90 up_thru 0 down_at 87 last_clean_interval [77,86) [v2:127.0.0.1:6874/3406014930,v1:127.0.0.1:6875/3406014930] [v2:127.0.0.1:6876/3406014930,v1:127.0.0.1:6877/3406014930] exists,up fc630115-1953-4e3d-885f-d2824414ef72 2024-06-03T06:53:09.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:53:09.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:53:09.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:53:09.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:53:09.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:53:09.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:53:09.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:53:09.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:53:09.010 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:53:09.010 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:53:09.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:53:09.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:53:09.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:53:09.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:53:09.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:53:09.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:53:09.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:53:09.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:53:09.107 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:53:09.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:53:09.394 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:53:09.395 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:53:09.395 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:53:09.395 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-06-03T06:53:09.395 INFO:tasks.workunit.client.0.smithi039.stderr:5 2024-06-03T06:53:09.395 INFO:tasks.workunit.client.0.smithi039.stderr:6 2024-06-03T06:53:09.395 INFO:tasks.workunit.client.0.smithi039.stderr:7 2024-06-03T06:53:09.395 INFO:tasks.workunit.client.0.smithi039.stderr:8 2024-06-03T06:53:09.395 INFO:tasks.workunit.client.0.smithi039.stderr:9' 2024-06-03T06:53:09.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:53:09.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:09.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:53:09.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836571 2024-06-03T06:53:09.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836571 2024-06-03T06:53:09.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571' 2024-06-03T06:53:09.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:09.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:53:09.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673043 2024-06-03T06:53:09.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673043 2024-06-03T06:53:09.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673043' 2024-06-03T06:53:09.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:09.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:53:09.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509517 2024-06-03T06:53:09.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509517 2024-06-03T06:53:09.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673043 2-64424509517' 2024-06-03T06:53:09.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:09.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:53:10.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678031 2024-06-03T06:53:10.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678031 2024-06-03T06:53:10.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673043 2-64424509517 3-304942678031' 2024-06-03T06:53:10.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:10.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:53:10.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182460 2024-06-03T06:53:10.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182460 2024-06-03T06:53:10.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673043 2-64424509517 3-304942678031 4-107374182460' 2024-06-03T06:53:10.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:10.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:53:10.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252871 2024-06-03T06:53:10.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252871 2024-06-03T06:53:10.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673043 2-64424509517 3-304942678031 4-107374182460 5-360777252871' 2024-06-03T06:53:10.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:10.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:53:10.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855410 2024-06-03T06:53:10.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855410 2024-06-03T06:53:10.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673043 2-64424509517 3-304942678031 4-107374182460 5-360777252871 6-150323855410' 2024-06-03T06:53:10.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:10.859 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:53:11.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691881 2024-06-03T06:53:11.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691881 2024-06-03T06:53:11.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673043 2-64424509517 3-304942678031 4-107374182460 5-360777252871 6-150323855410 7-171798691881' 2024-06-03T06:53:11.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:11.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:53:11.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528360 2024-06-03T06:53:11.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528360 2024-06-03T06:53:11.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673043 2-64424509517 3-304942678031 4-107374182460 5-360777252871 6-150323855410 7-171798691881 8-193273528360' 2024-06-03T06:53:11.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:11.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:53:11.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056642 2024-06-03T06:53:11.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056642 2024-06-03T06:53:11.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673043 2-64424509517 3-304942678031 4-107374182460 5-360777252871 6-150323855410 7-171798691881 8-193273528360 9-386547056642' 2024-06-03T06:53:11.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:11.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836571 2024-06-03T06:53:11.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:11.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:53:11.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836571 2024-06-03T06:53:11.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:11.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836571 2024-06-03T06:53:11.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-06-03T06:53:11.346 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836571 2024-06-03T06:53:11.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:53:11.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836571 -lt 21474836571 2024-06-03T06:53:11.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:11.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673043 2024-06-03T06:53:11.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:11.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:53:11.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673043 2024-06-03T06:53:11.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:11.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673043 2024-06-03T06:53:11.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-06-03T06:53:11.639 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949673043 2024-06-03T06:53:11.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:53:11.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673043 -lt 42949673043 2024-06-03T06:53:11.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:11.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509517 2024-06-03T06:53:11.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:11.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:53:11.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509517 2024-06-03T06:53:11.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:11.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509517 2024-06-03T06:53:11.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509517' 2024-06-03T06:53:11.973 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509517 2024-06-03T06:53:11.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:53:12.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509517 -lt 64424509517 2024-06-03T06:53:12.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:12.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678031 2024-06-03T06:53:12.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:12.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:53:12.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678031 2024-06-03T06:53:12.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:12.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678031 2024-06-03T06:53:12.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678031' 2024-06-03T06:53:12.265 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.3 seq 304942678031 2024-06-03T06:53:12.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:53:12.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678031 -lt 304942678031 2024-06-03T06:53:12.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:12.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182460 2024-06-03T06:53:12.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:12.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:53:12.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182460 2024-06-03T06:53:12.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:12.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182460 2024-06-03T06:53:12.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182460' 2024-06-03T06:53:12.592 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.4 seq 107374182460 2024-06-03T06:53:12.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:53:12.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182460 -lt 107374182460 2024-06-03T06:53:12.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:12.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252871 2024-06-03T06:53:12.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:12.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:53:12.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252871 2024-06-03T06:53:12.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:12.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252871 2024-06-03T06:53:12.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252871' 2024-06-03T06:53:12.887 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.5 seq 360777252871 2024-06-03T06:53:12.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:53:13.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252871 -lt 360777252871 2024-06-03T06:53:13.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:13.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855410 2024-06-03T06:53:13.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:13.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:53:13.195 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855410 2024-06-03T06:53:13.195 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:13.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855410 2024-06-03T06:53:13.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855410' 2024-06-03T06:53:13.195 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.6 seq 150323855410 2024-06-03T06:53:13.195 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:53:13.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855411 -lt 150323855410 2024-06-03T06:53:13.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:13.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691881 2024-06-03T06:53:13.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:13.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:53:13.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691881 2024-06-03T06:53:13.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:13.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691881 2024-06-03T06:53:13.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691881' 2024-06-03T06:53:13.511 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.7 seq 171798691881 2024-06-03T06:53:13.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:53:13.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691881 -lt 171798691881 2024-06-03T06:53:13.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:13.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528360 2024-06-03T06:53:13.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:13.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:53:13.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528360 2024-06-03T06:53:13.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:13.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528360 2024-06-03T06:53:13.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528360' 2024-06-03T06:53:13.836 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.8 seq 193273528360 2024-06-03T06:53:13.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:53:14.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528360 -lt 193273528360 2024-06-03T06:53:14.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:14.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-386547056642 2024-06-03T06:53:14.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:14.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:53:14.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-386547056642 2024-06-03T06:53:14.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:14.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056642 2024-06-03T06:53:14.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 386547056642' 2024-06-03T06:53:14.163 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.9 seq 386547056642 2024-06-03T06:53:14.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:53:14.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056642 -lt 386547056642 2024-06-03T06:53:14.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:53:14.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:53:14.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:53:14.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:53:14.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:53:14.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:53:14.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:53:14.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:53:14.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:53:14.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:53:14.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:53:15.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:53:15.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:53:15.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:53:15.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:53:15.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:53:15.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:53:15.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:53:15.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:53:15.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:53:15.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-06-03T06:53:15.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-03T06:53:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-03T06:53:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-06-03T06:53:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-06-03T06:53:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-03T06:53:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:53:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:53:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:53:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 268788"' 2024-06-03T06:53:15.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 268788' 2024-06-03T06:53:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-06-03T06:53:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:53:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:53:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:53:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 268790"' 2024-06-03T06:53:15.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 268790' 2024-06-03T06:53:15.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/268791: /' 2024-06-03T06:53:15.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-03T06:53:15.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 268788 268790' 2024-06-03T06:53:15.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:53:15.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:53:15.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 268788 2024-06-03T06:53:15.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-06-03T06:53:15.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/268794: /' 2024-06-03T06:53:18.862 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:53:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:53:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-03T06:53:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:53:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:53:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:53:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:53:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-03T06:53:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:53:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-03T06:53:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:53:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:53:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:53:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:53:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:53:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:53:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:53:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:53:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:53:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-03T06:53:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:53:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:53:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-03T06:53:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:268791: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-03T06:53:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-03T06:53:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:53:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:53:18.865 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-03T06:53:18.865 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:53:18.865 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:53:18.865 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:53:18.865 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:53:18.866 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:53:18.866 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:53:18.866 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-03T06:53:18.866 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-03T06:53:18.866 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:53:18.866 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:53:18.866 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:53:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:53:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:53:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:53:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:268791: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:53:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:268791: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:53:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:268791: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:53:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:53:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:53:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:53:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:53:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:53:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:53:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:53:18.868 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:53:18.868 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:53:18.868 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:53:18.868 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:53:18.868 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-03T06:53:18.868 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-03T06:53:18.868 INFO:tasks.workunit.client.0.smithi039.stderr:268791: start osd.5 2024-06-03T06:53:18.868 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:53:18.897 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:53:18.897 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:53:18.897 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-03T06:53:18.897 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:53:18.897 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-06-03T06:53:18.897 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:53:18.897 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:53:18.897 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-03T06:53:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:53:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-03T06:53:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:53:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:53:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:53:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:53:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:53:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:53:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-06-03T06:53:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:53:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:53:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-03T06:53:18.899 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:53:18.899 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:53:18.899 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-06-03T06:53:18.899 INFO:tasks.workunit.client.0.smithi039.stderr:268794: remove 2#2:eb822e21:::SOMETHING:head# 2024-06-03T06:53:18.899 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-03T06:53:18.899 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:53:18.899 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:53:18.899 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-03T06:53:18.899 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:53:18.899 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:53:18.899 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:53:18.899 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:53:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:53:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:53:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-03T06:53:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-03T06:53:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:53:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:53:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:53:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:53:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:53:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:53:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:268794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:53:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:268794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:53:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:268794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:53:18.902 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:53:18.902 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:53:18.902 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:53:18.902 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:53:18.902 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:53:18.902 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:53:18.902 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:53:18.902 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:53:18.902 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:53:18.903 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:53:18.903 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:53:18.903 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-03T06:53:18.903 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-03T06:53:18.903 INFO:tasks.workunit.client.0.smithi039.stderr:268794: start osd.9 2024-06-03T06:53:18.903 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:53:23.641 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-03T06:53:23.642 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-03T06:53:23.642 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:53:23.642 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:53:23.642 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:53:23.642 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 2024-06-03T06:53:18.931+0000 7f0042e1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:53:23.642 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 2024-06-03T06:53:18.934+0000 7f0042e1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:53:23.642 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 2024-06-03T06:53:18.935+0000 7f0042e1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:53:23.642 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-03T06:53:23.642 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:53:23.642 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-03T06:53:23.642 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:53:23.642 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:53:23.643 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:23.643 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:53:23.643 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 0 2024-06-03T06:53:23.643 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:23.643 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:53:23.643 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 2024-06-03T06:53:19.459+0000 7f0042e1f640 -1 Falling back to public interface 2024-06-03T06:53:23.643 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:23.643 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:23.643 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:23.643 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:53:23.643 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 1 2024-06-03T06:53:23.643 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:23.643 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:53:23.644 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:23.644 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:23.644 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:23.644 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:53:23.644 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 2 2024-06-03T06:53:23.644 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:23.644 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:53:23.644 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:23.644 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 2024-06-03T06:53:22.293+0000 7f0042e1f640 -1 osd.9 92 log_to_monitors true 2024-06-03T06:53:23.644 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:23.644 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:23.644 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:53:23.645 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 3 2024-06-03T06:53:23.645 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:23.645 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:53:23.645 INFO:tasks.workunit.client.0.smithi039.stderr:268794: osd.9 up in weight 1 up_from 97 up_thru 0 down_at 93 last_c268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-03T06:53:23.645 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-03T06:53:23.645 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:53:23.645 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:53:23.645 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:53:23.645 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 2024-06-03T06:53:18.897+0000 7f5c3ce0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:53:23.645 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 2024-06-03T06:53:18.900+0000 7f5c3ce0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:53:23.645 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 2024-06-03T06:53:18.902+0000 7f5c3ce0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:53:23.645 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-03T06:53:23.645 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:53:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-03T06:53:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:53:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:53:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:53:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 0 2024-06-03T06:53:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:53:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 2024-06-03T06:53:19.417+0000 7f5c3ce0e640 -1 Falling back to public interface 2024-06-03T06:53:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:53:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 1 2024-06-03T06:53:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:53:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:53:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 2 2024-06-03T06:53:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:53:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 2024-06-03T06:53:22.074+0000 7f5c3ce0e640 -1 osd.5 92 log_to_monitors true 2024-06-03T06:53:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:23.648 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:23.648 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:53:23.648 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 3 2024-06-03T06:53:23.648 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:23.648 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:53:24.995 INFO:tasks.workunit.client.0.smithi039.stderr:268791: osd.5 up in weight 1 up_from 97 up_thru 84 down_at 93 last_clean_interval [84,92) [v2:127.0.0.1:6842/4267692084,v1:127.0.0.1:6843/4267692084] [v2:127.0.0.1:6844/4267692084,v1:127.0.0.1:6845/4267692084] exists,up 915fce46-b35c-47e4-a164-8e162d251809 2024-06-03T06:53:24.995 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:53:24.995 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:53:24.995 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:53:24.995 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:53:24.995 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:53:24.995 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:53:24.995 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:53:24.995 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:53:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:268791: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:53:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:268791: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:53:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:53:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:53:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:53:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:53:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:53:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:53:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:53:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:53:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:53:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:53:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 1 2024-06-03T06:53:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 2 2024-06-03T06:53:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 3 2024-06-03T06:53:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 4 2024-06-03T06:53:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 5 2024-06-03T06:53:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 6 2024-06-03T06:53:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 7 2024-06-03T06:53:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 8 2024-06-03T06:53:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:268791: 9' 2024-06-03T06:53:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:53:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:53:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836576 2024-06-03T06:53:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836576 2024-06-03T06:53:24.998 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576' 2024-06-03T06:53:24.998 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:24.998 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:53:24.998 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673048 2024-06-03T06:53:24.998 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673048 2024-06-03T06:53:24.998 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673048' 2024-06-03T06:53:24.998 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:24.998 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qlean_interval [90,92) [v2:127.0.0.1:6874/1332734221,v1:127.0.0.1:6875/1332734221] [v2:127.0.0.1:6876/1332734221,v1:127.0.0.1:6877/1332734221] exists,up fc630115-1953-4e3d-885f-d2824414ef72 2024-06-03T06:53:24.998 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:53:24.998 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:53:24.998 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:53:24.998 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:53:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:53:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:53:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:53:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:53:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:268794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:53:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:268794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:53:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:53:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:53:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:53:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:53:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:53:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:53:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:53:25.000 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:53:25.000 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:53:25.000 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:53:25.000 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 1 2024-06-03T06:53:25.000 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 2 2024-06-03T06:53:25.000 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 3 2024-06-03T06:53:25.000 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 4 2024-06-03T06:53:25.000 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 5 2024-06-03T06:53:25.000 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 6 2024-06-03T06:53:25.000 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 7 2024-06-03T06:53:25.000 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 8 2024-06-03T06:53:25.000 INFO:tasks.workunit.client.0.smithi039.stderr:268794: 9' 2024-06-03T06:53:25.000 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:53:25.000 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:25.001 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:53:25.001 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836575 2024-06-03T06:53:25.001 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836575 2024-06-03T06:53:25.001 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575' 2024-06-03T06:53:25.001 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:25.001 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:53:25.001 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673047 2024-06-03T06:53:25.001 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673047 2024-06-03T06:53:25.001 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673047' 2024-06-03T06:53:25.001 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:26.230 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:53:26.230 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509521 2024-06-03T06:53:26.230 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509521 2024-06-03T06:53:26.230 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673047 2-64424509521' 2024-06-03T06:53:26.230 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:26.230 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:53:26.231 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678036 2024-06-03T06:53:26.231 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678036 2024-06-03T06:53:26.231 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673047 2-64424509521 3-304942678036' 2024-06-03T06:53:26.231 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:26.231 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:53:26.231 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182465 2024-06-03T06:53:26.231 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182465 2024-06-03T06:53:26.231 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673047 2-64424509521 3-304942678036 4-107374182465' 2024-06-03T06:53:26.231 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:26.231 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:53:26.231 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827715 2024-06-03T06:53:26.231 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827715 2024-06-03T06:53:26.231 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673047 2-64424509521 3-304942678036 4-107374182465 5-416611827715' 2024-06-03T06:53:26.231 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:26.232 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:53:26.232 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855415 2024-06-03T06:53:26.232 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855415 2024-06-03T06:53:26.232 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673047 2-64424509521 3-304942678036 4-107374182465 5-416611827715 6-150323855415' 2024-06-03T06:53:26.232 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:26.232 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:53:26.232 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691885 2024-06-03T06:53:26.232 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691885 2024-06-03T06:53:26.232 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673047 2-64424509521 3-304942678036 4-107374182465 5-416611827715 6-150323855415 7-171798691885' 2024-06-03T06:53:26.232 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:26.232 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:53:26.232 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clonea/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:53:26.232 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509522 2024-06-03T06:53:26.233 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509522 2024-06-03T06:53:26.233 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509522' 2024-06-03T06:53:26.233 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:26.233 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:53:26.233 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678035 2024-06-03T06:53:26.233 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678035 2024-06-03T06:53:26.233 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509522 3-304942678035' 2024-06-03T06:53:26.233 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:26.233 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:53:26.233 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182464 2024-06-03T06:53:26.233 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182464 2024-06-03T06:53:26.233 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509522 3-304942678035 4-107374182464' 2024-06-03T06:53:26.234 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:26.234 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:53:26.234 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827714 2024-06-03T06:53:26.234 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827714 2024-06-03T06:53:26.234 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509522 3-304942678035 4-107374182464 5-416611827714' 2024-06-03T06:53:26.234 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:26.234 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:53:26.234 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855414 2024-06-03T06:53:26.234 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855414 2024-06-03T06:53:26.234 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509522 3-304942678035 4-107374182464 5-416611827714 6-150323855414' 2024-06-03T06:53:26.234 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:26.234 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:53:26.234 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691886 2024-06-03T06:53:26.235 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691886 2024-06-03T06:53:26.235 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509522 3-304942678035 4-107374182464 5-416611827714 6-150323855414 7-171798691886' 2024-06-03T06:53:26.235 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:26.235 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:53:26.960 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528365 2024-06-03T06:53:26.960 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528365 2024-06-03T06:53:26.960 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509522 3-304942678035 4-107374182464 5-416611827714 6-150323855414 7-171798691886 8-193273528365' 2024-06-03T06:53:26.960 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:26.960 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:53:26.960 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827715 2024-06-03T06:53:26.960 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827715 2024-06-03T06:53:26.960 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509522 3-304942678035 4-107374182464 5-416611827714 6-150323855414 7-171798691886 8-193273528365 9-416611827715' 2024-06-03T06:53:26.961 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:26.961 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836576 2024-06-03T06:53:26.961 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:26.961 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:53:26.961 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836576 2024-06-03T06:53:26.961 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:26.961 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836576 2024-06-03T06:53:26.961 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836576' 2024-06-03T06:53:26.961 INFO:tasks.workunit.client.0.smithi039.stderr:268791: waiting osd.0 seq 21474836576 2024-06-03T06:53:26.961 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:53:26.961 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836576 2024-06-03T06:53:26.961 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:26.962 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673048 2024-06-03T06:53:26.962 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:26.962 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:53:26.962 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673048 2024-06-03T06:53:26.962 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:26.962 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673048 2024-06-03T06:53:26.962 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673048' 2024-06-03T06:53:26.962 INFO:tasks.workunit.client.0.smithi039.stderr:268791: waiting osd.1 seq 42949673048 2024-06-03T06:53:26.962 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:53:26.962 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673048 -lt 42949673048 2024-06-03T06:53:26.962 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:26.962 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509522 2024-06-03T06:53:26.962 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:26.962 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:53:27.010 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone..client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528364 2024-06-03T06:53:27.010 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528364 2024-06-03T06:53:27.010 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673047 2-64424509521 3-304942678036 4-107374182465 5-416611827715 6-150323855415 7-171798691885 8-193273528364' 2024-06-03T06:53:27.010 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:27.011 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:53:27.011 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827714 2024-06-03T06:53:27.011 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827714 2024-06-03T06:53:27.011 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673047 2-64424509521 3-304942678036 4-107374182465 5-416611827715 6-150323855415 7-171798691885 8-193273528364 9-416611827714' 2024-06-03T06:53:27.011 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:27.011 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836575 2024-06-03T06:53:27.011 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:27.011 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:53:27.011 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836575 2024-06-03T06:53:27.011 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:27.011 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836575 2024-06-03T06:53:27.011 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2024-06-03T06:53:27.011 INFO:tasks.workunit.client.0.smithi039.stderr:268794: waiting osd.0 seq 21474836575 2024-06-03T06:53:27.012 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:53:27.012 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836575 2024-06-03T06:53:27.012 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:27.012 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673047 2024-06-03T06:53:27.012 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:27.012 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:53:27.012 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673047 2024-06-03T06:53:27.012 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:27.012 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673047 2024-06-03T06:53:27.012 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673047' 2024-06-03T06:53:27.012 INFO:tasks.workunit.client.0.smithi039.stderr:268794: waiting osd.1 seq 42949673047 2024-06-03T06:53:27.012 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:53:27.012 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673048 -lt 42949673047 2024-06-03T06:53:27.012 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:27.013 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509521 2024-06-03T06:53:27.013 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:27.013 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:53:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.cclient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509522 2024-06-03T06:53:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509522 2024-06-03T06:53:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509522' 2024-06-03T06:53:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:268791: waiting osd.2 seq 64424509522 2024-06-03T06:53:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:53:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509522 -lt 64424509522 2024-06-03T06:53:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678035 2024-06-03T06:53:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:27.918 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:53:27.918 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678035 2024-06-03T06:53:27.918 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:27.918 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678035 2024-06-03T06:53:27.918 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678035' 2024-06-03T06:53:27.918 INFO:tasks.workunit.client.0.smithi039.stderr:268791: waiting osd.3 seq 304942678035 2024-06-03T06:53:27.918 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:53:27.918 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678037 -lt 304942678035 2024-06-03T06:53:27.918 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:27.918 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182464 2024-06-03T06:53:27.918 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:27.918 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:53:27.919 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182464 2024-06-03T06:53:27.919 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:27.919 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182464 2024-06-03T06:53:27.919 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182464' 2024-06-03T06:53:27.919 INFO:tasks.workunit.client.0.smithi039.stderr:268791: waiting osd.4 seq 107374182464 2024-06-03T06:53:27.919 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:53:27.919 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182466 -lt 107374182464 2024-06-03T06:53:27.919 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:27.919 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-416611827714 2024-06-03T06:53:27.919 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:27.919 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:53:27.919 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-416611827714 2024-06-03T06:53:27.919 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:27.920 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827714 2024-06-03T06:53:27.959 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitlient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509521 2024-06-03T06:53:27.959 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:27.959 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509521 2024-06-03T06:53:27.959 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-06-03T06:53:27.960 INFO:tasks.workunit.client.0.smithi039.stderr:268794: waiting osd.2 seq 64424509521 2024-06-03T06:53:27.960 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:53:27.960 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509522 -lt 64424509521 2024-06-03T06:53:27.960 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:27.960 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678036 2024-06-03T06:53:27.960 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:27.960 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:53:27.960 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678036 2024-06-03T06:53:27.960 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:27.960 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678036 2024-06-03T06:53:27.960 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678036' 2024-06-03T06:53:27.960 INFO:tasks.workunit.client.0.smithi039.stderr:268794: waiting osd.3 seq 304942678036 2024-06-03T06:53:27.961 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:53:27.961 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678037 -lt 304942678036 2024-06-03T06:53:27.961 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:27.961 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182465 2024-06-03T06:53:27.961 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:27.961 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:53:27.961 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182465 2024-06-03T06:53:27.961 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:27.961 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182465 2024-06-03T06:53:27.961 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182465' 2024-06-03T06:53:27.961 INFO:tasks.workunit.client.0.smithi039.stderr:268794: waiting osd.4 seq 107374182465 2024-06-03T06:53:27.961 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:53:27.961 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182466 -lt 107374182465 2024-06-03T06:53:27.962 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:27.962 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-416611827715 2024-06-03T06:53:27.962 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:27.962 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:53:27.962 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-416611827715 2024-06-03T06:53:27.962 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:27.962 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827715 2024-06-03T06:53:29.143 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitiing osd.5 seq 416611827714' 2024-06-03T06:53:29.143 INFO:tasks.workunit.client.0.smithi039.stderr:268791: waiting osd.5 seq 416611827714 2024-06-03T06:53:29.143 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:53:29.143 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827715 -lt 416611827714 2024-06-03T06:53:29.143 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:29.143 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855414 2024-06-03T06:53:29.143 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:29.144 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:53:29.144 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855414 2024-06-03T06:53:29.144 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:29.144 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855414 2024-06-03T06:53:29.144 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855414' 2024-06-03T06:53:29.144 INFO:tasks.workunit.client.0.smithi039.stderr:268791: waiting osd.6 seq 150323855414 2024-06-03T06:53:29.144 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:53:29.144 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855416 -lt 150323855414 2024-06-03T06:53:29.144 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:29.144 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691886 2024-06-03T06:53:29.144 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:29.144 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:53:29.144 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691886 2024-06-03T06:53:29.144 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691886 2024-06-03T06:53:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691886' 2024-06-03T06:53:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:268791: waiting osd.7 seq 171798691886 2024-06-03T06:53:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:53:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691886 -lt 171798691886 2024-06-03T06:53:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528365 2024-06-03T06:53:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:53:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528365 2024-06-03T06:53:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528365 2024-06-03T06:53:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528365' 2024-06-03T06:53:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:268791: waiting osd.8 seq 193273528365 2024-06-03T06:53:29.146 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:53:29.146 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528365 -lt 193273528365 2024-06-03T06:53:29.233 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_png osd.5 seq 416611827715' 2024-06-03T06:53:29.233 INFO:tasks.workunit.client.0.smithi039.stderr:268794: waiting osd.5 seq 416611827715 2024-06-03T06:53:29.233 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:53:29.233 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827715 -lt 416611827715 2024-06-03T06:53:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855415 2024-06-03T06:53:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:53:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855415 2024-06-03T06:53:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855415 2024-06-03T06:53:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855415' 2024-06-03T06:53:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:268794: waiting osd.6 seq 150323855415 2024-06-03T06:53:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:53:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855416 -lt 150323855415 2024-06-03T06:53:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691885 2024-06-03T06:53:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:29.235 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:53:29.235 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691885 2024-06-03T06:53:29.235 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:29.235 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691885 2024-06-03T06:53:29.235 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691885' 2024-06-03T06:53:29.235 INFO:tasks.workunit.client.0.smithi039.stderr:268794: waiting osd.7 seq 171798691885 2024-06-03T06:53:29.235 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:53:29.235 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691886 -lt 171798691885 2024-06-03T06:53:29.235 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:29.235 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528364 2024-06-03T06:53:29.235 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:29.235 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:53:29.236 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528364 2024-06-03T06:53:29.236 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:29.236 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528364 2024-06-03T06:53:29.236 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528364' 2024-06-03T06:53:29.236 INFO:tasks.workunit.client.0.smithi039.stderr:268794: waiting osd.8 seq 193273528364 2024-06-03T06:53:29.236 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:53:29.236 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528365 -lt 193273528364 2024-06-03T06:53:30.505 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pgg_stats: for s in $seqs 2024-06-03T06:53:30.505 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-416611827715 2024-06-03T06:53:30.505 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:30.505 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:53:30.505 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-416611827715 2024-06-03T06:53:30.506 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:30.506 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827715 2024-06-03T06:53:30.506 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 416611827715' 2024-06-03T06:53:30.506 INFO:tasks.workunit.client.0.smithi039.stderr:268791: waiting osd.9 seq 416611827715 2024-06-03T06:53:30.506 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:53:30.507 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827716 -lt 416611827715 2024-06-03T06:53:30.507 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:53:30.507 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:53:30.507 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:53:30.507 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:53:30.508 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:53:30.508 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:53:30.508 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:53:30.508 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:53:30.508 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:53:30.508 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:53:30.509 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:53:30.509 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:53:30.509 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:53:30.509 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:53:30.509 INFO:tasks.workunit.client.0.smithi039.stderr:268791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:53:30.510 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:53:30.510 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:53:30.510 INFO:tasks.workunit.client.0.smithi039.stderr:268791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:53:30.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:53:30.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:53:30.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 268790 2024-06-03T06:53:30.676 INFO:tasks.workunit.client.0.smithi039.stderr:_stats: for s in $seqs 2024-06-03T06:53:30.677 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-416611827714 2024-06-03T06:53:30.677 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:30.677 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:53:30.677 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-416611827714 2024-06-03T06:53:30.677 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:30.677 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827714 2024-06-03T06:53:30.677 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 416611827714' 2024-06-03T06:53:30.677 INFO:tasks.workunit.client.0.smithi039.stderr:268794: waiting osd.9 seq 416611827714 2024-06-03T06:53:30.677 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:53:30.678 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827716 -lt 416611827714 2024-06-03T06:53:30.678 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:53:30.678 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:53:30.678 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:53:30.678 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:53:30.678 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:53:30.678 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:53:30.678 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:53:30.678 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:53:30.678 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:53:30.678 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:53:30.678 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:53:30.679 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:53:30.679 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:53:30.679 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:53:30.679 INFO:tasks.workunit.client.0.smithi039.stderr:268794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:53:30.679 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:53:30.679 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:53:30.679 INFO:tasks.workunit.client.0.smithi039.stderr:268794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:53:30.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:53:30.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:53:30.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:53:30.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:53:30.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-03T06:53:30.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:53:30.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-03T06:53:30.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:53:30.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:53:30.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:53:30.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:53:30.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-03T06:53:30.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-03T06:53:30.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:53:30.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:53:30.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:53:30.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:53:30.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:53:30.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:53:31.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:52:56.787674+0000 2024-06-03T06:53:31.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:53:31.567 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:53:31.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:52:56.787674+0000 2024-06-03T06:53:31.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:53:31.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:52:56.787674+0000 2024-06-03T06:53:31.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:53:31.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:53:31.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:53:31.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:53:31.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:53:31.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:53:31.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:53:31.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:53:31.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:52:56.787674+0000 '>' 2024-06-03T06:52:56.787674+0000 2024-06-03T06:53:31.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:53:32.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:53:32.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:53:32.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:53:32.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:53:32.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:53:32.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:53:32.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:53:33.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:52:56.787674+0000 '>' 2024-06-03T06:52:56.787674+0000 2024-06-03T06:53:33.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:53:34.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:53:34.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:53:34.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:53:34.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:53:34.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:53:34.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:53:34.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:53:34.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:52:56.787674+0000 '>' 2024-06-03T06:52:56.787674+0000 2024-06-03T06:53:34.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:53:35.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:53:35.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:53:35.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:53:35.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:53:35.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:53:35.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:53:35.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:53:35.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:52:56.787674+0000 '>' 2024-06-03T06:52:56.787674+0000 2024-06-03T06:53:35.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:53:36.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:53:36.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:53:36.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:53:36.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:53:36.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:53:36.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:53:36.777 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:53:37.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:53:31.773256+0000 '>' 2024-06-03T06:52:56.787674+0000 2024-06-03T06:53:37.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:53:37.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-03T06:53:37.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:53:37.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:53:37.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:53:37.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 272194"' 2024-06-03T06:53:37.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 272194' 2024-06-03T06:53:37.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:53:37.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:53:37.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:53:37.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 272196"' 2024-06-03T06:53:37.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 272196' 2024-06-03T06:53:37.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:53:37.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-03T06:53:37.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 272194 272196' 2024-06-03T06:53:37.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:53:37.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:53:37.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 272194 2024-06-03T06:53:37.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/272197: /' 2024-06-03T06:53:37.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:53:37.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/272199: /' 2024-06-03T06:53:39.969 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:53:39.969 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:53:39.970 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-03T06:53:39.970 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:53:39.970 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:53:39.970 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:53:39.970 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:53:39.970 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-03T06:53:39.970 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:53:39.970 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-03T06:53:39.970 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:53:39.970 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:53:39.970 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:53:39.970 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:53:39.970 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:53:39.971 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:53:39.971 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-03T06:53:39.971 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:53:39.971 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:53:39.971 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-03T06:53:39.971 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:53:39.971 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:53:39.971 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-03T06:53:39.971 INFO:tasks.workunit.client.0.smithi039.stderr:272199: _ 2024-06-03T06:53:39.971 INFO:tasks.workunit.client.0.smithi039.stderr:272199: hinfo_key 2024-06-03T06:53:39.971 INFO:tasks.workunit.client.0.smithi039.stderr:272199: snapset 2024-06-03T06:53:39.971 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-03T06:53:39.971 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:53:39.972 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:53:39.972 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-03T06:53:39.972 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:53:39.972 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-03T06:53:39.972 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:53:39.972 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:53:39.973 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:53:39.973 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:53:39.973 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-03T06:53:39.973 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-03T06:53:39.973 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:53:39.973 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:53:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:53:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:53:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:53:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:53:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:272199: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:53:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:272199: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:53:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:272199: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:53:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:53:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:53:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:53:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:53:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:53:39.975 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:53:39.975 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:53:39.975 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:53:39.975 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:53:39.975 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:53:39.975 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:53:39.975 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-03T06:53:39.975 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-03T06:53:39.975 INFO:tasks.workunit.client.0.smithi039.stderr:272199: start osd.9 2024-06-03T06:53:40.104 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:53:40.104 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:53:40.104 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-03T06:53:40.104 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:53:40.104 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:53:40.105 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:53:40.105 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:53:40.105 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-03T06:53:40.105 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:53:40.105 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-03T06:53:40.105 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:53:40.105 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:53:40.105 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:53:40.105 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:53:40.105 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:53:40.105 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:53:40.105 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:53:40.105 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:53:40.106 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:53:40.106 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-03T06:53:40.106 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:53:40.106 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:53:40.106 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-06-03T06:53:40.106 INFO:tasks.workunit.client.0.smithi039.stderr:272197: _ 2024-06-03T06:53:40.106 INFO:tasks.workunit.client.0.smithi039.stderr:272197: hinfo_key 2024-06-03T06:53:40.106 INFO:tasks.workunit.client.0.smithi039.stderr:272197: snapset 2024-06-03T06:53:40.106 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-03T06:53:40.106 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:53:40.106 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:53:40.106 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-03T06:53:40.107 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:53:40.107 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:53:40.107 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:53:40.107 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:53:40.107 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:53:40.107 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:53:40.107 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-03T06:53:40.107 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-03T06:53:40.107 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:53:40.107 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:53:40.108 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:53:40.108 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:53:40.108 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:53:40.108 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:53:40.108 INFO:tasks.workunit.client.0.smithi039.stderr:272197: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:53:40.108 INFO:tasks.workunit.client.0.smithi039.stderr:272197: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:53:40.108 INFO:tasks.workunit.client.0.smithi039.stderr:272197: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:53:40.108 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:53:40.108 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:53:40.108 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:53:40.108 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:53:40.108 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:53:40.108 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:53:40.109 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:53:40.109 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:53:40.109 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:53:40.109 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:53:40.109 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:53:40.109 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-03T06:53:40.109 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-03T06:53:40.109 INFO:tasks.workunit.client.0.smithi039.stderr:272197: start osd.5 2024-06-03T06:53:44.696 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:53:44.696 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-03T06:53:44.696 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-03T06:53:44.696 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:53:44.696 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:53:44.696 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:53:44.696 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 2024-06-03T06:53:40.135+0000 7f846e813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:53:44.696 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 2024-06-03T06:53:40.137+0000 7f846e813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:53:44.696 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 2024-06-03T06:53:40.138+0000 7f846e813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:53:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-03T06:53:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:53:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-03T06:53:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:53:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:53:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:53:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 0 2024-06-03T06:53:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:53:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 2024-06-03T06:53:40.654+0000 7f846e813640 -1 Falling back to public interface 2024-06-03T06:53:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:44.698 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:53:44.698 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 1 2024-06-03T06:53:44.698 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:44.698 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:53:44.698 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:44.698 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:44.698 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:44.698 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:53:44.698 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 2 2024-06-03T06:53:44.698 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:44.698 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:53:44.698 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:44.698 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 2024-06-03T06:53:43.452+0000 7f846e813640 -1 osd.5 99 log_to_monitors true 2024-06-03T06:53:44.699 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:44.699 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:44.699 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:53:44.699 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 3 2024-06-03T06:53:44.699 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:44.699 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:53:44.699 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/covery_ops 2024-06-03T06:53:44.699 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-03T06:53:44.699 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-03T06:53:44.699 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:53:44.699 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:53:44.699 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:53:44.699 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 2024-06-03T06:53:40.004+0000 7f98d651e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:53:44.700 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 2024-06-03T06:53:40.006+0000 7f98d651e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:53:44.700 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 2024-06-03T06:53:40.007+0000 7f98d651e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:53:44.700 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-03T06:53:44.700 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:53:44.700 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-03T06:53:44.700 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:53:44.700 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:53:44.700 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:44.700 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:53:44.700 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 0 2024-06-03T06:53:44.700 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:44.700 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:53:44.700 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 2024-06-03T06:53:40.528+0000 7f98d651e640 -1 Falling back to public interface 2024-06-03T06:53:44.701 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:44.701 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:44.701 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:44.701 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:53:44.701 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 1 2024-06-03T06:53:44.701 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:44.701 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:53:44.701 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:44.701 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:44.701 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:44.701 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:53:44.701 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 2 2024-06-03T06:53:44.701 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:44.701 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:53:44.702 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 2024-06-03T06:53:43.169+0000 7f98d651e640 -1 osd.9 99 log_to_monitors true 2024-06-03T06:53:44.702 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:44.702 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:44.702 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:44.702 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:53:44.702 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 3 2024-06-03T06:53:44.702 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:44.702 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:53:46.929 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:46.929 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:46.929 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:46.929 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:53:46.929 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 4 2024-06-03T06:53:46.929 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:46.929 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:53:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:272197: osd.5 up in weight 1 up_from 104 up_thru 104 down_at 100 last_clean_interval [97,99) [v2:127.0.0.1:6874/300260261,v1:127.0.0.1:6875/300260261] [v2:127.0.0.1:6876/300260261,v1:127.0.0.1:6877/300260261] exists,up 915fce46-b35c-47e4-a164-8e162d251809 2024-06-03T06:53:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:53:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:53:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:53:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:53:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:53:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:53:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:53:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:53:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:272197: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:53:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:272197: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:53:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:53:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:53:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:53:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:53:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:53:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:53:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:53:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:53:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:53:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:53:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 1 2024-06-03T06:53:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 2 2024-06-03T06:53:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 3 2024-06-03T06:53:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 4 2024-06-03T06:53:46.932 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 5 2024-06-03T06:53:46.932 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 6 2024-06-03T06:53:46.932 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 7 2024-06-03T06:53:46.932 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 8 2024-06-03T06:53:46.932 INFO:tasks.workunit.client.0.smithi039.stderr:272197: 9' 2024-06-03T06:53:46.932 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:53:46.932 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:46.932 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:53:46.932 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836581 2024-06-03T06:53:46.932 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836581 2024-06-03T06:53:46.932 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581' 2024-06-03T06:53:46.932 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $idseph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:53:46.932 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:53:46.932 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:53:46.933 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:53:46.933 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 4 2024-06-03T06:53:46.933 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:53:46.933 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-03T06:53:46.933 INFO:tasks.workunit.client.0.smithi039.stderr:272199: osd.9 up in weight 1 up_from 104 up_thru 0 down_at 100 last_clean_interval [97,99) [v2:127.0.0.1:6842/1939328475,v1:127.0.0.1:6843/1939328475] [v2:127.0.0.1:6844/1939328475,v1:127.0.0.1:6845/1939328475] exists,up fc630115-1953-4e3d-885f-d2824414ef72 2024-06-03T06:53:46.933 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:53:46.933 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:53:46.933 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:53:46.933 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:53:46.933 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:53:46.933 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:53:46.933 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:53:46.933 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:53:46.933 INFO:tasks.workunit.client.0.smithi039.stderr:272199: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:53:46.933 INFO:tasks.workunit.client.0.smithi039.stderr:272199: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:53:46.934 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:53:46.934 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:53:46.934 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:53:46.934 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:53:46.934 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:53:46.934 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:53:46.934 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:53:46.934 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:53:46.934 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:53:46.934 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:53:46.934 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 1 2024-06-03T06:53:46.934 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 2 2024-06-03T06:53:46.935 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 3 2024-06-03T06:53:46.935 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 4 2024-06-03T06:53:46.935 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 5 2024-06-03T06:53:46.935 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 6 2024-06-03T06:53:46.935 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 7 2024-06-03T06:53:46.935 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 8 2024-06-03T06:53:46.935 INFO:tasks.workunit.client.0.smithi039.stderr:272199: 9' 2024-06-03T06:53:46.935 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:53:46.935 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:46.935 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:53:46.935 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836582 2024-06-03T06:53:46.935 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836582 2024-06-03T06:53:46.935 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582' 2024-06-03T06:53:47.786 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $i 2024-06-03T06:53:47.786 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:53:47.786 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673054 2024-06-03T06:53:47.786 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673054 2024-06-03T06:53:47.786 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054' 2024-06-03T06:53:47.786 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:47.787 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:53:47.787 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509528 2024-06-03T06:53:47.787 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509528 2024-06-03T06:53:47.787 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509528' 2024-06-03T06:53:47.787 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:47.787 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:53:47.787 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678043 2024-06-03T06:53:47.787 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678043 2024-06-03T06:53:47.787 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509528 3-304942678043' 2024-06-03T06:53:47.787 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:47.788 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:53:47.788 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182472 2024-06-03T06:53:47.788 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182472 2024-06-03T06:53:47.788 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509528 3-304942678043 4-107374182472' 2024-06-03T06:53:47.789 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:47.789 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:53:47.789 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598787 2024-06-03T06:53:47.789 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598787 2024-06-03T06:53:47.789 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509528 3-304942678043 4-107374182472 5-446676598787' 2024-06-03T06:53:47.789 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:47.789 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:53:47.790 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855422 2024-06-03T06:53:47.790 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855422 2024-06-03T06:53:47.790 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509528 3-304942678043 4-107374182472 5-446676598787 6-150323855422' 2024-06-03T06:53:47.790 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:47.790 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:53:47.790 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:22ds 2024-06-03T06:53:47.790 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:53:47.790 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673053 2024-06-03T06:53:47.790 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673053 2024-06-03T06:53:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673053' 2024-06-03T06:53:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:53:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509529 2024-06-03T06:53:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509529 2024-06-03T06:53:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673053 2-64424509529' 2024-06-03T06:53:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:53:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678042 2024-06-03T06:53:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678042 2024-06-03T06:53:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673053 2-64424509529 3-304942678042' 2024-06-03T06:53:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:47.791 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:53:47.792 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182471 2024-06-03T06:53:47.792 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182471 2024-06-03T06:53:47.792 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673053 2-64424509529 3-304942678042 4-107374182471' 2024-06-03T06:53:47.792 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:47.792 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:53:47.792 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598786 2024-06-03T06:53:47.792 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598786 2024-06-03T06:53:47.792 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673053 2-64424509529 3-304942678042 4-107374182471 5-446676598786' 2024-06-03T06:53:47.792 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:47.792 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:53:47.793 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855421 2024-06-03T06:53:47.793 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855421 2024-06-03T06:53:47.793 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673053 2-64424509529 3-304942678042 4-107374182471 5-446676598786 6-150323855421' 2024-06-03T06:53:47.793 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:47.793 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:53:48.335 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691892 2024-06-03T06:53:48.336 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691892 2024-06-03T06:53:48.336 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673053 2-64424509529 3-304942678042 4-107374182471 5-446676598786 6-150323855421 7-171798691892' 2024-06-03T06:53:48.336 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:48.336 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:53:48.336 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528370 2024-06-03T06:53:48.336 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528370 2024-06-03T06:53:48.336 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673053 2-64424509529 3-304942678042 4-107374182471 5-446676598786 6-150323855421 7-171798691892 8-193273528370' 2024-06-03T06:53:48.336 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:48.336 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:53:48.336 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598786 2024-06-03T06:53:48.336 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598786 2024-06-03T06:53:48.336 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673053 2-64424509529 3-304942678042 4-107374182471 5-446676598786 6-150323855421 7-171798691892 8-193273528370 9-446676598786' 2024-06-03T06:53:48.337 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:48.337 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836582 2024-06-03T06:53:48.337 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:48.337 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:53:48.337 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836582 2024-06-03T06:53:48.337 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:48.337 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836582 2024-06-03T06:53:48.337 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2024-06-03T06:53:48.337 INFO:tasks.workunit.client.0.smithi039.stderr:272199: waiting osd.0 seq 21474836582 2024-06-03T06:53:48.337 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:53:48.337 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836582 -lt 21474836582 2024-06-03T06:53:48.337 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:48.337 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673053 2024-06-03T06:53:48.338 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:48.338 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:53:48.338 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673053 2024-06-03T06:53:48.338 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:48.338 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673053 2024-06-03T06:53:48.338 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673053' 2024-06-03T06:53:48.338 INFO:tasks.workunit.client.0.smithi039.stderr:272199: waiting osd.1 seq 42949673053 2024-06-03T06:53:48.340 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221402: flush_pg_stats: seq=171798691891 2024-06-03T06:53:48.341 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691891 2024-06-03T06:53:48.341 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509528 3-304942678043 4-107374182472 5-446676598787 6-150323855422 7-171798691891' 2024-06-03T06:53:48.341 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:48.341 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:53:48.341 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528371 2024-06-03T06:53:48.341 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528371 2024-06-03T06:53:48.341 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509528 3-304942678043 4-107374182472 5-446676598787 6-150323855422 7-171798691891 8-193273528371' 2024-06-03T06:53:48.341 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:53:48.341 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:53:48.341 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598787 2024-06-03T06:53:48.341 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598787 2024-06-03T06:53:48.341 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509528 3-304942678043 4-107374182472 5-446676598787 6-150323855422 7-171798691891 8-193273528371 9-446676598787' 2024-06-03T06:53:48.341 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:48.341 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836581 2024-06-03T06:53:48.342 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:48.342 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:53:48.342 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836581 2024-06-03T06:53:48.342 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:48.342 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836581 2024-06-03T06:53:48.342 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-06-03T06:53:48.342 INFO:tasks.workunit.client.0.smithi039.stderr:272197: waiting osd.0 seq 21474836581 2024-06-03T06:53:48.342 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:53:48.342 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836582 -lt 21474836581 2024-06-03T06:53:48.342 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:48.342 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673054 2024-06-03T06:53:48.342 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:48.342 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:53:48.342 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673054 2024-06-03T06:53:48.343 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:48.343 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673054 2024-06-03T06:53:48.343 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-06-03T06:53:48.343 INFO:tasks.workunit.client.0.smithi039.stderr:272197: waiting osd.1 seq 42949673054 2024-06-03T06:53:50.600 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: : flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:53:50.601 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673053 2024-06-03T06:53:50.601 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:53:50.601 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:53:50.601 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:53:50.601 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673055 -lt 42949673053 2024-06-03T06:53:50.601 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:50.601 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509529 2024-06-03T06:53:50.601 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:50.602 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:53:50.602 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509529 2024-06-03T06:53:50.602 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:50.602 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509529 2024-06-03T06:53:50.602 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-06-03T06:53:50.602 INFO:tasks.workunit.client.0.smithi039.stderr:272199: waiting osd.2 seq 64424509529 2024-06-03T06:53:50.602 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:53:50.602 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509529 -lt 64424509529 2024-06-03T06:53:50.602 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:50.602 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678042 2024-06-03T06:53:50.602 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:50.602 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:53:50.603 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678042 2024-06-03T06:53:50.603 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:50.603 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678042 2024-06-03T06:53:50.603 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678042' 2024-06-03T06:53:50.603 INFO:tasks.workunit.client.0.smithi039.stderr:272199: waiting osd.3 seq 304942678042 2024-06-03T06:53:50.603 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:53:50.603 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678043 -lt 304942678042 2024-06-03T06:53:50.603 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:50.603 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182471 2024-06-03T06:53:50.603 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:50.603 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:53:50.603 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182471 2024-06-03T06:53:50.603 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:50.604 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182471 2024-06-03T06:53:50.604 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182471' 2024-06-03T06:53:50.604 INFO:tasks.workunit.client.0.smithi039.stderr:272199: waiting osd.4 seq 107374182471 2024-06-03T06:53:50.632 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/ceflush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:53:50.632 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673054 2024-06-03T06:53:50.632 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:53:50.632 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:53:50.632 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:53:50.632 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673055 -lt 42949673054 2024-06-03T06:53:50.632 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:50.632 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509528 2024-06-03T06:53:50.633 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:50.633 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:53:50.633 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509528 2024-06-03T06:53:50.633 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:50.633 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509528 2024-06-03T06:53:50.633 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509528' 2024-06-03T06:53:50.633 INFO:tasks.workunit.client.0.smithi039.stderr:272197: waiting osd.2 seq 64424509528 2024-06-03T06:53:50.633 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:53:50.633 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509529 -lt 64424509528 2024-06-03T06:53:50.633 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:50.633 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678043 2024-06-03T06:53:50.633 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:50.633 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:53:50.633 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678043 2024-06-03T06:53:50.634 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:50.634 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678043 2024-06-03T06:53:50.634 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678043' 2024-06-03T06:53:50.634 INFO:tasks.workunit.client.0.smithi039.stderr:272197: waiting osd.3 seq 304942678043 2024-06-03T06:53:50.634 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:53:50.634 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678043 -lt 304942678043 2024-06-03T06:53:50.634 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:50.634 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182472 2024-06-03T06:53:50.634 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:50.634 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:53:50.634 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182472 2024-06-03T06:53:50.634 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:50.634 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182472 2024-06-03T06:53:50.635 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182472' 2024-06-03T06:53:50.635 INFO:tasks.workunit.client.0.smithi039.stderr:272197: waiting osd.4 seq 107374182472 2024-06-03T06:53:51.838 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:53:51.838 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182472 -lt 107374182472 2024-06-03T06:53:51.838 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:51.838 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-446676598787 2024-06-03T06:53:51.838 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:51.838 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:53:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-446676598787 2024-06-03T06:53:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598787 2024-06-03T06:53:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 446676598787' 2024-06-03T06:53:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:272197: waiting osd.5 seq 446676598787 2024-06-03T06:53:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:53:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598788 -lt 446676598787 2024-06-03T06:53:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855422 2024-06-03T06:53:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:53:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855422 2024-06-03T06:53:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:51.839 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855422 2024-06-03T06:53:51.840 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855422' 2024-06-03T06:53:51.840 INFO:tasks.workunit.client.0.smithi039.stderr:272197: waiting osd.6 seq 150323855422 2024-06-03T06:53:51.840 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:53:51.840 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855422 -lt 150323855422 2024-06-03T06:53:51.840 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:51.840 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691891 2024-06-03T06:53:51.840 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:51.840 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:53:51.840 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691891 2024-06-03T06:53:51.840 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:51.840 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691891 2024-06-03T06:53:51.840 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691891' 2024-06-03T06:53:51.840 INFO:tasks.workunit.client.0.smithi039.stderr:272197: waiting osd.7 seq 171798691891 2024-06-03T06:53:51.841 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:53:51.841 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691892 -lt 171798691891 2024-06-03T06:53:51.841 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:51.881 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:53:51.881 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182472 -lt 107374182471 2024-06-03T06:53:51.881 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:51.881 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-446676598786 2024-06-03T06:53:51.881 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:51.881 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:53:51.881 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-446676598786 2024-06-03T06:53:51.881 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:51.881 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598786 2024-06-03T06:53:51.882 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 446676598786' 2024-06-03T06:53:51.882 INFO:tasks.workunit.client.0.smithi039.stderr:272199: waiting osd.5 seq 446676598786 2024-06-03T06:53:51.882 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:53:51.882 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598788 -lt 446676598786 2024-06-03T06:53:51.882 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:51.882 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855421 2024-06-03T06:53:51.882 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:51.882 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:53:51.882 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855421 2024-06-03T06:53:51.882 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:51.882 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855421 2024-06-03T06:53:51.882 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855421' 2024-06-03T06:53:51.882 INFO:tasks.workunit.client.0.smithi039.stderr:272199: waiting osd.6 seq 150323855421 2024-06-03T06:53:51.883 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:53:51.883 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855422 -lt 150323855421 2024-06-03T06:53:51.883 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:51.883 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691892 2024-06-03T06:53:51.883 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:51.883 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:53:51.883 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691892 2024-06-03T06:53:51.883 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:51.883 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691892 2024-06-03T06:53:51.883 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691892' 2024-06-03T06:53:51.884 INFO:tasks.workunit.client.0.smithi039.stderr:272199: waiting osd.7 seq 171798691892 2024-06-03T06:53:51.884 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:53:51.884 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691892 -lt 171798691892 2024-06-03T06:53:51.884 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:53.128 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hpers.sh:2211: flush_pg_stats: echo 8-193273528371 2024-06-03T06:53:53.128 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:53.128 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:53:53.128 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528371 2024-06-03T06:53:53.129 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:53.129 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528371 2024-06-03T06:53:53.129 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528371' 2024-06-03T06:53:53.129 INFO:tasks.workunit.client.0.smithi039.stderr:272197: waiting osd.8 seq 193273528371 2024-06-03T06:53:53.129 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:53:53.129 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528372 -lt 193273528371 2024-06-03T06:53:53.129 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:53.129 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-446676598787 2024-06-03T06:53:53.129 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:53.129 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:53:53.129 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-446676598787 2024-06-03T06:53:53.129 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:53.130 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598787 2024-06-03T06:53:53.130 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 446676598787' 2024-06-03T06:53:53.130 INFO:tasks.workunit.client.0.smithi039.stderr:272197: waiting osd.9 seq 446676598787 2024-06-03T06:53:53.130 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:53:53.130 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598788 -lt 446676598787 2024-06-03T06:53:53.130 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:53:53.130 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:53:53.130 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:53:53.130 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:53:53.130 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:53:53.130 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:53:53.130 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:53:53.131 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:53:53.131 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:53:53.131 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:53:53.131 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:53:53.131 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:53:53.131 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:53:53.179 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph -elpers.sh:2211: flush_pg_stats: echo 8-193273528370 2024-06-03T06:53:53.179 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:53.179 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:53:53.179 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528370 2024-06-03T06:53:53.180 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:53.180 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528370 2024-06-03T06:53:53.180 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528370' 2024-06-03T06:53:53.180 INFO:tasks.workunit.client.0.smithi039.stderr:272199: waiting osd.8 seq 193273528370 2024-06-03T06:53:53.180 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:53:53.180 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528372 -lt 193273528370 2024-06-03T06:53:53.180 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:53:53.180 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-446676598786 2024-06-03T06:53:53.180 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:53:53.180 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:53:53.180 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-446676598786 2024-06-03T06:53:53.180 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:53:53.180 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598786 2024-06-03T06:53:53.181 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 446676598786' 2024-06-03T06:53:53.181 INFO:tasks.workunit.client.0.smithi039.stderr:272199: waiting osd.9 seq 446676598786 2024-06-03T06:53:53.181 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:53:53.181 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598788 -lt 446676598786 2024-06-03T06:53:53.181 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:53:53.181 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:53:53.181 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:53:53.181 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:53:53.181 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:53:53.181 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:53:53.181 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:53:53.182 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:53:53.182 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:53:53.182 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:53:53.182 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:53:53.182 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:53:53.182 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:53:53.505 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph-format json status 2024-06-03T06:53:53.505 INFO:tasks.workunit.client.0.smithi039.stderr:272197: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:53:53.506 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:53:53.506 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:53:53.506 INFO:tasks.workunit.client.0.smithi039.stderr:272197: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:53:53.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:53:53.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:53:53.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 272196 2024-06-03T06:53:53.552 INFO:tasks.workunit.client.0.smithi039.stderr: --format json status 2024-06-03T06:53:53.552 INFO:tasks.workunit.client.0.smithi039.stderr:272199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:53:53.553 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:53:53.553 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:53:53.553 INFO:tasks.workunit.client.0.smithi039.stderr:272199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:53:53.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:53:53.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:53:53.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:53:53.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:53:53.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-03T06:53:53.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:53:53.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:53:53.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:53:53.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-06-03T06:53:53.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-03T06:53:53.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-03T06:53:53.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-06-03T06:53:53.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-06-03T06:53:53.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-03T06:53:53.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:53:53.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:53:53.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:53:53.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 275521"' 2024-06-03T06:53:53.605 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 275521' 2024-06-03T06:53:53.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:53:53.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:53:53.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:53:53.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:53:53.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 275523"' 2024-06-03T06:53:53.605 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 275523' 2024-06-03T06:53:53.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/275524: /' 2024-06-03T06:53:53.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-03T06:53:53.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 275521 275523' 2024-06-03T06:53:53.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:53:53.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:53:53.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 275521 2024-06-03T06:53:53.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:53:53.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/275526: /' 2024-06-03T06:53:57.395 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:53:57.396 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:53:57.396 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:53:57.396 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:53:57.396 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:53:57.396 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:53:57.396 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:53:57.396 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:53:57.396 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:53:57.396 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:53:57.396 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:53:57.396 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:53:57.396 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:53:57.396 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:53:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:53:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:53:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-03T06:53:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:53:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:53:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:53:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:53:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:53:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-03T06:53:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:275524: remove 0#2:eb822e21:::SOMETHING:head# 2024-06-03T06:53:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:53:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:53:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:53:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:53:57.397 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:53:57.398 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:53:57.398 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:53:57.398 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:53:57.399 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:53:57.399 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:53:57.399 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:53:57.399 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:53:57.399 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:53:57.399 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:53:57.399 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:53:57.399 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:53:57.400 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:53:57.400 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:53:57.400 INFO:tasks.workunit.client.0.smithi039.stderr:275524: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:53:57.400 INFO:tasks.workunit.client.0.smithi039.stderr:275524: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:53:57.400 INFO:tasks.workunit.client.0.smithi039.stderr:275524: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:53:57.400 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:53:57.400 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:53:57.400 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:53:57.400 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:53:57.400 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:53:57.400 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:53:57.400 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:53:57.400 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:53:57.400 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:53:57.401 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:53:57.401 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:53:57.401 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:53:57.401 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:53:57.401 INFO:tasks.workunit.client.0.smithi039.stderr:275524: start osd.3 2024-06-03T06:53:57.401 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:53:57.602 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:53:57.602 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:53:57.602 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-03T06:53:57.602 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:53:57.602 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:53:57.602 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:53:57.602 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:53:57.602 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-03T06:53:57.602 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:53:57.603 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-03T06:53:57.603 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:53:57.603 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:53:57.603 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:53:57.603 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:53:57.603 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:53:57.603 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:53:57.603 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-03T06:53:57.603 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:53:57.603 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:53:57.603 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-03T06:53:57.603 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:53:57.603 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:53:57.603 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-03T06:53:57.604 INFO:tasks.workunit.client.0.smithi039.stderr:275526: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-03T06:53:57.604 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-03T06:53:57.604 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:53:57.604 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:53:57.604 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-03T06:53:57.604 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:53:57.604 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:53:57.604 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:53:57.604 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:53:57.605 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:53:57.605 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:53:57.605 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-03T06:53:57.605 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-03T06:53:57.605 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:53:57.605 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:53:57.605 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:53:57.605 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:53:57.605 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:53:57.605 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:53:57.605 INFO:tasks.workunit.client.0.smithi039.stderr:275526: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:53:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:275526: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:53:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:275526: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:53:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:53:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:53:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:53:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:53:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:53:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:53:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:53:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:53:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:53:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:53:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:53:57.607 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-03T06:53:57.607 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-03T06:53:57.607 INFO:tasks.workunit.client.0.smithi039.stderr:275526: start osd.5 2024-06-03T06:53:57.607 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:54:02.267 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:54:02.268 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:54:02.268 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:54:02.268 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:54:02.268 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:54:02.268 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 2024-06-03T06:53:57.430+0000 7f880162e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:02.268 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 2024-06-03T06:53:57.432+0000 7f880162e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:02.268 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 2024-06-03T06:53:57.434+0000 7f880162e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:02.268 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:54:02.268 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:54:02.268 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:54:02.268 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:54:02.269 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:54:02.269 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:02.269 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:54:02.269 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 0 2024-06-03T06:54:02.269 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:02.269 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:54:02.269 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 2024-06-03T06:53:57.956+0000 7f880162e640 -1 Falling back to public interface 2024-06-03T06:54:02.269 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:02.269 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:02.269 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:02.269 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:54:02.269 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 1 2024-06-03T06:54:02.269 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:02.269 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:54:02.269 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:02.270 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:02.270 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:02.270 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:54:02.270 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 2 2024-06-03T06:54:02.270 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:02.270 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:54:02.270 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 2024-06-03T06:54:00.619+0000 7f880162e640 -1 osd.3 107 log_to_monitors true 2024-06-03T06:54:02.270 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:02.270 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:02.270 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:02.270 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:54:02.270 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 3 2024-06-03T06:54:02.270 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:02.271 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:54:02.652 INFO:tasks.workunit.client.0.smithi039.stderr:275524: osd.3 up in weight 1 up_from 112 up_thru 112 down_at 107 l275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-03T06:54:02.652 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-03T06:54:02.652 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:54:02.652 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:54:02.652 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:54:02.652 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 2024-06-03T06:53:57.632+0000 7fc05e807640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:02.652 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 2024-06-03T06:53:57.634+0000 7fc05e807640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:02.652 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 2024-06-03T06:53:57.635+0000 7fc05e807640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:02.652 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-03T06:54:02.652 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:54:02.652 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-03T06:54:02.652 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:54:02.652 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:54:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:54:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 0 2024-06-03T06:54:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:54:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 2024-06-03T06:53:58.158+0000 7fc05e807640 -1 Falling back to public interface 2024-06-03T06:54:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:54:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 1 2024-06-03T06:54:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:54:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:54:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 2 2024-06-03T06:54:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:54:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 2024-06-03T06:54:01.048+0000 7fc05e807640 -1 osd.5 107 log_to_monitors true 2024-06-03T06:54:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:54:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 3 2024-06-03T06:54:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:54:03.557 INFO:tasks.workunit.client.0.smithi039.stderr:275526: osd.5 up in weight 1 up_from 113 up_thru 104 down_at 108 last_clean_interval [71,107) [v2:127.0.0.1:6826/3792558713,v1:127.0.0.1:6827/3792558713] [v2:127.0.0.1:6828/3792558713,v1:127.0.0.1:6829/3792558713] exists,up 79666bc5-58ed-418d-906d-594dd41a932e 2024-06-03T06:54:03.557 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:54:03.557 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:54:03.557 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:54:03.557 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:54:03.557 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:54:03.558 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:54:03.558 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:54:03.558 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:54:03.558 INFO:tasks.workunit.client.0.smithi039.stderr:275524: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:54:03.558 INFO:tasks.workunit.client.0.smithi039.stderr:275524: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:54:03.558 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:54:03.558 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:54:03.558 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:54:03.558 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:54:03.558 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:54:03.558 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:54:03.558 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:54:03.558 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:54:03.559 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:54:03.559 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:54:03.559 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 1 2024-06-03T06:54:03.559 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 2 2024-06-03T06:54:03.559 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 3 2024-06-03T06:54:03.559 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 4 2024-06-03T06:54:03.559 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 5 2024-06-03T06:54:03.559 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 6 2024-06-03T06:54:03.559 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 7 2024-06-03T06:54:03.559 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 8 2024-06-03T06:54:03.559 INFO:tasks.workunit.client.0.smithi039.stderr:275524: 9' 2024-06-03T06:54:03.559 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:54:03.559 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:03.559 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:54:03.560 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836586 2024-06-03T06:54:03.560 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836586 2024-06-03T06:54:03.560 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586' 2024-06-03T06:54:03.560 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:03.560 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:54:03.560 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673059 2024-06-03T06:54:03.560 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673059 2024-06-03T06:54:03.560 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673059' 2024-06-03T06:54:03.560 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:03.831 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.clienast_clean_interval [104,107) [v2:127.0.0.1:6874/1726175390,v1:127.0.0.1:6875/1726175390] [v2:127.0.0.1:6876/1726175390,v1:127.0.0.1:6877/1726175390] exists,up 915fce46-b35c-47e4-a164-8e162d251809 2024-06-03T06:54:03.831 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:54:03.831 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:54:03.832 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:54:03.832 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:54:03.832 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:54:03.832 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:54:03.832 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:54:03.832 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:54:03.832 INFO:tasks.workunit.client.0.smithi039.stderr:275526: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:54:03.832 INFO:tasks.workunit.client.0.smithi039.stderr:275526: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:54:03.832 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:54:03.832 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:54:03.832 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:54:03.832 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:54:03.833 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:54:03.833 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:54:03.833 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:54:03.833 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:54:03.833 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:54:03.833 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:54:03.833 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 1 2024-06-03T06:54:03.833 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 2 2024-06-03T06:54:03.833 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 3 2024-06-03T06:54:03.833 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 4 2024-06-03T06:54:03.833 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 5 2024-06-03T06:54:03.833 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 6 2024-06-03T06:54:03.833 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 7 2024-06-03T06:54:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 8 2024-06-03T06:54:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:275526: 9' 2024-06-03T06:54:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:54:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:54:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836587 2024-06-03T06:54:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836587 2024-06-03T06:54:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587' 2024-06-03T06:54:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:54:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673060 2024-06-03T06:54:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673060 2024-06-03T06:54:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673060' 2024-06-03T06:54:03.834 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:04.762 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.cliet.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:54:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509533 2024-06-03T06:54:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509533 2024-06-03T06:54:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673059 2-64424509533' 2024-06-03T06:54:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:54:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337154 2024-06-03T06:54:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337154 2024-06-03T06:54:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673059 2-64424509533 3-481036337154' 2024-06-03T06:54:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:54:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182476 2024-06-03T06:54:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182476 2024-06-03T06:54:04.764 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673059 2-64424509533 3-481036337154 4-107374182476' 2024-06-03T06:54:04.764 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:04.764 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:54:04.764 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304450 2024-06-03T06:54:04.764 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304450 2024-06-03T06:54:04.764 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673059 2-64424509533 3-481036337154 4-107374182476 5-485331304450' 2024-06-03T06:54:04.764 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:04.764 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:54:04.764 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855426 2024-06-03T06:54:04.764 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855426 2024-06-03T06:54:04.764 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673059 2-64424509533 3-481036337154 4-107374182476 5-485331304450 6-150323855426' 2024-06-03T06:54:04.764 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:04.764 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:54:04.765 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691896 2024-06-03T06:54:04.765 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691896 2024-06-03T06:54:04.765 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673059 2-64424509533 3-481036337154 4-107374182476 5-485331304450 6-150323855426 7-171798691896' 2024-06-03T06:54:04.765 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:04.765 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:54:04.873 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtestnt.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:54:04.873 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509534 2024-06-03T06:54:04.873 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509534 2024-06-03T06:54:04.873 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673060 2-64424509534' 2024-06-03T06:54:04.873 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:04.874 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:54:04.874 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337155 2024-06-03T06:54:04.874 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337155 2024-06-03T06:54:04.874 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673060 2-64424509534 3-481036337155' 2024-06-03T06:54:04.874 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:04.874 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:54:04.874 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182477 2024-06-03T06:54:04.874 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182477 2024-06-03T06:54:04.874 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673060 2-64424509534 3-481036337155 4-107374182477' 2024-06-03T06:54:04.874 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:04.874 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:54:04.874 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304451 2024-06-03T06:54:04.875 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304451 2024-06-03T06:54:04.875 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673060 2-64424509534 3-481036337155 4-107374182477 5-485331304451' 2024-06-03T06:54:04.875 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:04.875 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:54:04.875 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855427 2024-06-03T06:54:04.875 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855427 2024-06-03T06:54:04.875 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673060 2-64424509534 3-481036337155 4-107374182477 5-485331304451 6-150323855427' 2024-06-03T06:54:04.875 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:04.875 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:54:04.875 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691897 2024-06-03T06:54:04.875 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691897 2024-06-03T06:54:04.875 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673060 2-64424509534 3-481036337155 4-107374182477 5-485331304451 6-150323855427 7-171798691897' 2024-06-03T06:54:04.875 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:04.875 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:54:05.475 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtes/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528375 2024-06-03T06:54:05.475 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528375 2024-06-03T06:54:05.475 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673059 2-64424509533 3-481036337154 4-107374182476 5-485331304450 6-150323855426 7-171798691896 8-193273528375' 2024-06-03T06:54:05.475 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:05.475 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:54:05.475 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598792 2024-06-03T06:54:05.475 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598792 2024-06-03T06:54:05.475 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673059 2-64424509533 3-481036337154 4-107374182476 5-485331304450 6-150323855426 7-171798691896 8-193273528375 9-446676598792' 2024-06-03T06:54:05.476 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:05.476 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836586 2024-06-03T06:54:05.476 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:05.476 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:54:05.476 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836586 2024-06-03T06:54:05.476 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:05.476 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836586 2024-06-03T06:54:05.476 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2024-06-03T06:54:05.476 INFO:tasks.workunit.client.0.smithi039.stderr:275524: waiting osd.0 seq 21474836586 2024-06-03T06:54:05.476 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:54:05.476 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836588 -lt 21474836586 2024-06-03T06:54:05.476 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:05.477 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673059 2024-06-03T06:54:05.477 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:05.477 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:54:05.477 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673059 2024-06-03T06:54:05.477 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:05.477 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673059 2024-06-03T06:54:05.477 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-06-03T06:54:05.477 INFO:tasks.workunit.client.0.smithi039.stderr:275524: waiting osd.1 seq 42949673059 2024-06-03T06:54:05.477 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:54:05.477 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673060 -lt 42949673059 2024-06-03T06:54:05.477 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:05.477 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509533 2024-06-03T06:54:05.477 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:05.477 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:54:05.631 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/ct/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528376 2024-06-03T06:54:05.631 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528376 2024-06-03T06:54:05.632 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673060 2-64424509534 3-481036337155 4-107374182477 5-485331304451 6-150323855427 7-171798691897 8-193273528376' 2024-06-03T06:54:05.632 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:05.632 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:54:05.632 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598793 2024-06-03T06:54:05.632 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598793 2024-06-03T06:54:05.632 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673060 2-64424509534 3-481036337155 4-107374182477 5-485331304451 6-150323855427 7-171798691897 8-193273528376 9-446676598793' 2024-06-03T06:54:05.633 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:05.633 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836587 2024-06-03T06:54:05.633 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:05.633 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:54:05.633 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836587 2024-06-03T06:54:05.633 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:05.633 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836587 2024-06-03T06:54:05.634 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-06-03T06:54:05.634 INFO:tasks.workunit.client.0.smithi039.stderr:275526: waiting osd.0 seq 21474836587 2024-06-03T06:54:05.634 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:54:05.634 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836588 -lt 21474836587 2024-06-03T06:54:05.634 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:05.634 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673060 2024-06-03T06:54:05.634 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:05.635 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:54:05.635 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673060 2024-06-03T06:54:05.635 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:05.635 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673060 2024-06-03T06:54:05.635 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673060' 2024-06-03T06:54:05.635 INFO:tasks.workunit.client.0.smithi039.stderr:275526: waiting osd.1 seq 42949673060 2024-06-03T06:54:05.635 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:54:05.636 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673060 -lt 42949673060 2024-06-03T06:54:05.636 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:05.636 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509534 2024-06-03T06:54:05.636 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:05.636 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:54:06.436 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/lone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509533 2024-06-03T06:54:06.436 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:06.436 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509533 2024-06-03T06:54:06.436 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2024-06-03T06:54:06.436 INFO:tasks.workunit.client.0.smithi039.stderr:275524: waiting osd.2 seq 64424509533 2024-06-03T06:54:06.436 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:54:06.436 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509534 -lt 64424509533 2024-06-03T06:54:06.437 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:06.437 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-481036337154 2024-06-03T06:54:06.437 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:06.437 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:54:06.437 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-481036337154 2024-06-03T06:54:06.437 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:06.437 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337154 2024-06-03T06:54:06.437 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 481036337154' 2024-06-03T06:54:06.437 INFO:tasks.workunit.client.0.smithi039.stderr:275524: waiting osd.3 seq 481036337154 2024-06-03T06:54:06.437 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:54:06.437 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337155 -lt 481036337154 2024-06-03T06:54:06.437 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:06.437 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182476 2024-06-03T06:54:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:54:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182476 2024-06-03T06:54:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182476 2024-06-03T06:54:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182476' 2024-06-03T06:54:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:275524: waiting osd.4 seq 107374182476 2024-06-03T06:54:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:54:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182477 -lt 107374182476 2024-06-03T06:54:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-485331304450 2024-06-03T06:54:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:06.439 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:54:06.439 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-485331304450 2024-06-03T06:54:06.439 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:06.439 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304450 2024-06-03T06:54:06.591 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509534 2024-06-03T06:54:06.591 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:06.591 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509534 2024-06-03T06:54:06.591 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509534' 2024-06-03T06:54:06.592 INFO:tasks.workunit.client.0.smithi039.stderr:275526: waiting osd.2 seq 64424509534 2024-06-03T06:54:06.592 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:54:06.592 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509534 -lt 64424509534 2024-06-03T06:54:06.592 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:06.592 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-481036337155 2024-06-03T06:54:06.592 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:06.592 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:54:06.592 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-481036337155 2024-06-03T06:54:06.592 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:06.592 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337155 2024-06-03T06:54:06.592 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 481036337155' 2024-06-03T06:54:06.592 INFO:tasks.workunit.client.0.smithi039.stderr:275526: waiting osd.3 seq 481036337155 2024-06-03T06:54:06.592 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:54:06.593 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337155 -lt 481036337155 2024-06-03T06:54:06.593 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:06.593 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182477 2024-06-03T06:54:06.593 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:06.593 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:54:06.593 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182477 2024-06-03T06:54:06.593 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:06.593 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182477 2024-06-03T06:54:06.594 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182477' 2024-06-03T06:54:06.594 INFO:tasks.workunit.client.0.smithi039.stderr:275526: waiting osd.4 seq 107374182477 2024-06-03T06:54:06.594 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:54:06.594 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182477 -lt 107374182477 2024-06-03T06:54:06.594 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:06.594 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-485331304451 2024-06-03T06:54:06.594 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:06.594 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:54:06.594 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-485331304451 2024-06-03T06:54:06.594 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:06.594 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304451 2024-06-03T06:54:07.734 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo'waiting osd.5 seq 485331304450' 2024-06-03T06:54:07.734 INFO:tasks.workunit.client.0.smithi039.stderr:275524: waiting osd.5 seq 485331304450 2024-06-03T06:54:07.734 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:54:07.735 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304451 -lt 485331304450 2024-06-03T06:54:07.735 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:07.735 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855426 2024-06-03T06:54:07.735 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:07.735 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:54:07.735 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855426 2024-06-03T06:54:07.735 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:07.735 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855426 2024-06-03T06:54:07.735 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855426' 2024-06-03T06:54:07.735 INFO:tasks.workunit.client.0.smithi039.stderr:275524: waiting osd.6 seq 150323855426 2024-06-03T06:54:07.735 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:54:07.735 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855427 -lt 150323855426 2024-06-03T06:54:07.735 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:07.735 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691896 2024-06-03T06:54:07.736 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:07.736 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:54:07.736 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691896 2024-06-03T06:54:07.736 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:07.736 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691896 2024-06-03T06:54:07.736 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691896' 2024-06-03T06:54:07.736 INFO:tasks.workunit.client.0.smithi039.stderr:275524: waiting osd.7 seq 171798691896 2024-06-03T06:54:07.736 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:54:07.736 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691898 -lt 171798691896 2024-06-03T06:54:07.736 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:07.736 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528375 2024-06-03T06:54:07.736 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:07.737 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:54:07.737 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528375 2024-06-03T06:54:07.737 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:07.737 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528375 2024-06-03T06:54:07.737 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528375' 2024-06-03T06:54:07.737 INFO:tasks.workunit.client.0.smithi039.stderr:275524: waiting osd.8 seq 193273528375 2024-06-03T06:54:07.737 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:54:07.737 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528377 -lt 193273528375 2024-06-03T06:54:07.841 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fl 'waiting osd.5 seq 485331304451' 2024-06-03T06:54:07.841 INFO:tasks.workunit.client.0.smithi039.stderr:275526: waiting osd.5 seq 485331304451 2024-06-03T06:54:07.841 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:54:07.841 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304451 -lt 485331304451 2024-06-03T06:54:07.841 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:07.841 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855427 2024-06-03T06:54:07.841 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:07.841 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:54:07.841 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855427 2024-06-03T06:54:07.842 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:07.842 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855427 2024-06-03T06:54:07.842 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855427' 2024-06-03T06:54:07.842 INFO:tasks.workunit.client.0.smithi039.stderr:275526: waiting osd.6 seq 150323855427 2024-06-03T06:54:07.842 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:54:07.842 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855428 -lt 150323855427 2024-06-03T06:54:07.842 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:07.842 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691897 2024-06-03T06:54:07.842 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:07.842 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:54:07.842 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691897 2024-06-03T06:54:07.842 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:07.843 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691897 2024-06-03T06:54:07.843 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691897' 2024-06-03T06:54:07.843 INFO:tasks.workunit.client.0.smithi039.stderr:275526: waiting osd.7 seq 171798691897 2024-06-03T06:54:07.843 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:54:07.843 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691898 -lt 171798691897 2024-06-03T06:54:07.843 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:07.843 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528376 2024-06-03T06:54:07.843 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:07.843 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:54:07.843 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528376 2024-06-03T06:54:07.843 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:07.843 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528376 2024-06-03T06:54:07.843 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528376' 2024-06-03T06:54:07.844 INFO:tasks.workunit.client.0.smithi039.stderr:275526: waiting osd.8 seq 193273528376 2024-06-03T06:54:07.844 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:54:07.844 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528377 -lt 193273528376 2024-06-03T06:54:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fush_pg_stats: for s in $seqs 2024-06-03T06:54:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-446676598792 2024-06-03T06:54:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:54:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-446676598792 2024-06-03T06:54:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598792 2024-06-03T06:54:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 446676598792' 2024-06-03T06:54:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:275524: waiting osd.9 seq 446676598792 2024-06-03T06:54:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:54:09.130 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598793 -lt 446676598792 2024-06-03T06:54:09.130 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:54:09.130 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:54:09.130 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:54:09.130 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:54:09.131 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:54:09.131 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:54:09.131 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:54:09.131 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:54:09.131 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:54:09.132 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:54:09.132 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:54:09.132 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:54:09.132 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:54:09.132 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:54:09.133 INFO:tasks.workunit.client.0.smithi039.stderr:275524: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:54:09.133 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:54:09.133 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:54:09.133 INFO:tasks.workunit.client.0.smithi039.stderr:275524: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:54:09.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:54:09.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:54:09.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 275523 2024-06-03T06:54:09.180 INFO:tasks.workunit.client.0.smithi039.stderr:lush_pg_stats: for s in $seqs 2024-06-03T06:54:09.181 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-446676598793 2024-06-03T06:54:09.181 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:09.181 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:54:09.181 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-446676598793 2024-06-03T06:54:09.181 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:09.181 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598793 2024-06-03T06:54:09.181 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 446676598793' 2024-06-03T06:54:09.181 INFO:tasks.workunit.client.0.smithi039.stderr:275526: waiting osd.9 seq 446676598793 2024-06-03T06:54:09.181 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:54:09.181 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598793 -lt 446676598793 2024-06-03T06:54:09.181 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:54:09.182 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:54:09.182 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:54:09.182 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:54:09.182 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:54:09.182 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:54:09.182 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:54:09.182 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:54:09.182 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:54:09.182 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:54:09.183 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:54:09.183 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:54:09.183 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:54:09.183 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:54:09.183 INFO:tasks.workunit.client.0.smithi039.stderr:275526: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:54:09.183 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:54:09.183 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:54:09.183 INFO:tasks.workunit.client.0.smithi039.stderr:275526: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:54:09.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:54:09.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:54:09.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:54:09.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:54:09.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-03T06:54:09.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:54:09.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-03T06:54:09.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T06:54:09.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:54:09.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-03T06:54:09.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:54:09.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-03T06:54:09.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-03T06:54:09.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T06:54:09.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T06:54:09.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:54:09.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:54:09.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:54:09.475 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:54:09.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:53:31.773256+0000 2024-06-03T06:54:09.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T06:54:10.034 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-03T06:54:10.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T06:53:31.773256+0000 2024-06-03T06:54:10.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T06:54:10.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:53:31.773256+0000 2024-06-03T06:54:10.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:54:10.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:54:10.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:54:10.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:54:10.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:54:10.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:54:10.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:54:10.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:54:10.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:53:31.773256+0000 '>' 2024-06-03T06:53:31.773256+0000 2024-06-03T06:54:10.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:54:11.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:54:11.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:54:11.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:54:11.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:54:11.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:54:11.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:54:11.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:54:11.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:53:31.773256+0000 '>' 2024-06-03T06:53:31.773256+0000 2024-06-03T06:54:11.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:54:12.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:54:12.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:54:12.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:54:12.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:54:12.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:54:12.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:54:12.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:54:12.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:53:31.773256+0000 '>' 2024-06-03T06:53:31.773256+0000 2024-06-03T06:54:12.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:54:13.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:54:13.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:54:13.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T06:54:13.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T06:54:13.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:54:13.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:54:13.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T06:54:14.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:54:10.550159+0000 '>' 2024-06-03T06:53:31.773256+0000 2024-06-03T06:54:14.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:54:14.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-03T06:54:14.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:54:14.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:54:14.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:54:14.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 278903"' 2024-06-03T06:54:14.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 278903' 2024-06-03T06:54:14.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:54:14.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T06:54:14.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T06:54:14.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 278905"' 2024-06-03T06:54:14.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 278905' 2024-06-03T06:54:14.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-03T06:54:14.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:54:14.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 278903 278905' 2024-06-03T06:54:14.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T06:54:14.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:54:14.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 278903 2024-06-03T06:54:14.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/278906: /' 2024-06-03T06:54:14.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:54:14.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/278908: /' 2024-06-03T06:54:17.475 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:54:17.476 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:54:17.476 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-03T06:54:17.476 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:54:17.476 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:54:17.476 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:54:17.476 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:54:17.476 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-03T06:54:17.476 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:54:17.476 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-03T06:54:17.476 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:54:17.476 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:54:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:54:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:54:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:54:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:54:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-03T06:54:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:54:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:54:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-03T06:54:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:54:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:54:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-03T06:54:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:278906: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-03T06:54:17.477 INFO:tasks.workunit.client.0.smithi039.stderr:278906: _ 2024-06-03T06:54:17.478 INFO:tasks.workunit.client.0.smithi039.stderr:278906: hinfo_key 2024-06-03T06:54:17.478 INFO:tasks.workunit.client.0.smithi039.stderr:278906: snapset 2024-06-03T06:54:17.478 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-03T06:54:17.478 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:54:17.478 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:54:17.478 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-03T06:54:17.478 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:54:17.478 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-03T06:54:17.478 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:54:17.479 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:54:17.479 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:54:17.479 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:54:17.479 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-03T06:54:17.479 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-03T06:54:17.479 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:54:17.480 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:54:17.480 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:54:17.480 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:54:17.480 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:54:17.480 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:54:17.480 INFO:tasks.workunit.client.0.smithi039.stderr:278906: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:54:17.480 INFO:tasks.workunit.client.0.smithi039.stderr:278906: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:54:17.480 INFO:tasks.workunit.client.0.smithi039.stderr:278906: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:54:17.480 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:54:17.480 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:54:17.480 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:54:17.481 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:54:17.481 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:54:17.481 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:54:17.481 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:54:17.481 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:54:17.481 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:54:17.481 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:54:17.481 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:54:17.481 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-03T06:54:17.481 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-03T06:54:17.481 INFO:tasks.workunit.client.0.smithi039.stderr:278906: start osd.3 2024-06-03T06:54:17.715 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:54:17.715 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:54:17.715 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-03T06:54:17.715 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:54:17.715 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:54:17.715 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:54:17.715 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:54:17.715 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-03T06:54:17.715 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:54:17.715 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-03T06:54:17.716 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:54:17.716 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:54:17.716 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:54:17.716 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:54:17.716 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:54:17.716 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:54:17.716 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-03T06:54:17.716 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:54:17.716 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:54:17.716 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-03T06:54:17.716 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:54:17.716 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:54:17.716 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-06-03T06:54:17.717 INFO:tasks.workunit.client.0.smithi039.stderr:278908: _ 2024-06-03T06:54:17.717 INFO:tasks.workunit.client.0.smithi039.stderr:278908: hinfo_key 2024-06-03T06:54:17.717 INFO:tasks.workunit.client.0.smithi039.stderr:278908: snapset 2024-06-03T06:54:17.717 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-03T06:54:17.717 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:54:17.717 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:54:17.717 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-03T06:54:17.717 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:54:17.717 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-03T06:54:17.717 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:54:17.717 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:54:17.718 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:54:17.718 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:54:17.718 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-03T06:54:17.718 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-03T06:54:17.718 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:54:17.718 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:54:17.718 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:54:17.718 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:54:17.718 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:54:17.718 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:54:17.718 INFO:tasks.workunit.client.0.smithi039.stderr:278908: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:54:17.718 INFO:tasks.workunit.client.0.smithi039.stderr:278908: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:54:17.719 INFO:tasks.workunit.client.0.smithi039.stderr:278908: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:54:17.719 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:54:17.719 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:54:17.719 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:54:17.719 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:54:17.719 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:54:17.719 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:54:17.719 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:54:17.719 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:54:17.719 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:54:17.719 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:54:17.719 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:54:17.719 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-03T06:54:17.720 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-03T06:54:17.720 INFO:tasks.workunit.client.0.smithi039.stderr:278908: start osd.5 2024-06-03T06:54:21.718 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:54:21.718 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-03T06:54:21.718 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-03T06:54:21.718 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:54:21.719 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:54:21.719 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:54:21.719 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 2024-06-03T06:54:17.510+0000 7f03d48d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:21.719 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 2024-06-03T06:54:17.512+0000 7f03d48d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:21.719 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 2024-06-03T06:54:17.513+0000 7f03d48d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:21.719 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-03T06:54:21.719 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:54:21.719 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T06:54:21.719 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:54:21.719 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:54:21.719 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:21.720 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:54:21.720 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 0 2024-06-03T06:54:21.720 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:21.720 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:54:21.720 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 2024-06-03T06:54:18.039+0000 7f03d48d7640 -1 Falling back to public interface 2024-06-03T06:54:21.720 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:21.720 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:21.720 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:21.720 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:54:21.721 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 1 2024-06-03T06:54:21.721 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:21.721 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:54:21.721 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:21.721 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:21.721 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:21.721 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:54:21.721 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 2 2024-06-03T06:54:21.721 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:21.721 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:54:21.721 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:21.722 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 2024-06-03T06:54:20.725+0000 7f03d48d7640 -1 osd.3 115 log_to_monitors true 2024-06-03T06:54:21.722 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:21.722 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:21.722 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:54:21.722 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 3 2024-06-03T06:54:21.722 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:22.415 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-06-03T06:54:22.415 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-03T06:54:22.415 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-03T06:54:22.415 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:54:22.415 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:54:22.415 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:54:22.415 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 2024-06-03T06:54:17.744+0000 7f947ec1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:22.415 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 2024-06-03T06:54:17.747+0000 7f947ec1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:22.415 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 2024-06-03T06:54:17.748+0000 7f947ec1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:22.415 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-03T06:54:22.415 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:54:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-03T06:54:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:54:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:54:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:54:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 0 2024-06-03T06:54:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:54:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 2024-06-03T06:54:18.265+0000 7f947ec1a640 -1 Falling back to public interface 2024-06-03T06:54:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:54:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 1 2024-06-03T06:54:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:54:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:54:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 2 2024-06-03T06:54:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:54:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 2024-06-03T06:54:20.996+0000 7f947ec1a640 -1 osd.5 115 log_to_monitors true 2024-06-03T06:54:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:22.417 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:22.418 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:54:22.418 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 3 2024-06-03T06:54:22.418 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:22.418 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-03T06:54:23.030 INFO:tasks.workunit.client.0.smithi039.stderr:278908: osd.5 up in weight 1 up_from 120 up_thru 120 dowrs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T06:54:23.030 INFO:tasks.workunit.client.0.smithi039.stderr:278906: osd.3 up in weight 1 up_from 120 up_thru 113 down_at 116 last_clean_interval [112,115) [v2:127.0.0.1:6826/2999053824,v1:127.0.0.1:6827/2999053824] [v2:127.0.0.1:6828/2999053824,v1:127.0.0.1:6829/2999053824] exists,up 79666bc5-58ed-418d-906d-594dd41a932e 2024-06-03T06:54:23.030 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:54:23.030 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:54:23.030 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:54:23.030 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:54:23.031 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:54:23.031 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:54:23.031 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:54:23.031 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:54:23.031 INFO:tasks.workunit.client.0.smithi039.stderr:278906: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:54:23.031 INFO:tasks.workunit.client.0.smithi039.stderr:278906: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:54:23.031 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:54:23.031 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:54:23.031 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:54:23.031 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:54:23.031 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:54:23.031 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:54:23.032 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:54:23.032 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:54:23.032 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:54:23.032 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:54:23.032 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 1 2024-06-03T06:54:23.032 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 2 2024-06-03T06:54:23.032 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 3 2024-06-03T06:54:23.032 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 4 2024-06-03T06:54:23.032 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 5 2024-06-03T06:54:23.032 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 6 2024-06-03T06:54:23.032 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 7 2024-06-03T06:54:23.032 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 8 2024-06-03T06:54:23.032 INFO:tasks.workunit.client.0.smithi039.stderr:278906: 9' 2024-06-03T06:54:23.033 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:54:23.033 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:23.033 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:54:23.033 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836592 2024-06-03T06:54:23.033 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836592 2024-06-03T06:54:23.033 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592' 2024-06-03T06:54:23.033 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:23.033 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:54:23.033 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673065 2024-06-03T06:54:23.033 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673065 2024-06-03T06:54:23.033 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673065' 2024-06-03T06:54:23.158 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0n_at 116 last_clean_interval [113,115) [v2:127.0.0.1:6874/3331591231,v1:127.0.0.1:6875/3331591231] [v2:127.0.0.1:6876/3331591231,v1:127.0.0.1:6877/3331591231] exists,up 915fce46-b35c-47e4-a164-8e162d251809 2024-06-03T06:54:23.158 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:54:23.158 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:54:23.158 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:54:23.158 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:54:23.158 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:54:23.158 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:54:23.158 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:54:23.158 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:54:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:278908: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:54:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:278908: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:54:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:54:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:54:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:54:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:54:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:54:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:54:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:54:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:54:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:54:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:54:23.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 1 2024-06-03T06:54:23.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 2 2024-06-03T06:54:23.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 3 2024-06-03T06:54:23.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 4 2024-06-03T06:54:23.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 5 2024-06-03T06:54:23.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 6 2024-06-03T06:54:23.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 7 2024-06-03T06:54:23.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 8 2024-06-03T06:54:23.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: 9' 2024-06-03T06:54:23.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:54:23.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:23.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:54:23.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836593 2024-06-03T06:54:23.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836593 2024-06-03T06:54:23.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593' 2024-06-03T06:54:23.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:23.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:54:23.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673066 2024-06-03T06:54:23.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673066 2024-06-03T06:54:23.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673066' 2024-06-03T06:54:23.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:23.916 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest//qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:23.917 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:54:23.917 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509539 2024-06-03T06:54:23.917 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509539 2024-06-03T06:54:23.917 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673065 2-64424509539' 2024-06-03T06:54:23.917 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:23.917 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:54:23.917 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075522 2024-06-03T06:54:23.917 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075522 2024-06-03T06:54:23.917 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673065 2-64424509539 3-515396075522' 2024-06-03T06:54:23.917 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:23.917 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:54:23.918 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182482 2024-06-03T06:54:23.918 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182482 2024-06-03T06:54:23.918 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673065 2-64424509539 3-515396075522 4-107374182482' 2024-06-03T06:54:23.918 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:23.918 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:54:23.918 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075522 2024-06-03T06:54:23.918 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075522 2024-06-03T06:54:23.918 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673065 2-64424509539 3-515396075522 4-107374182482 5-515396075522' 2024-06-03T06:54:23.918 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:23.918 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:54:23.918 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855432 2024-06-03T06:54:23.919 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855432 2024-06-03T06:54:23.919 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673065 2-64424509539 3-515396075522 4-107374182482 5-515396075522 6-150323855432' 2024-06-03T06:54:23.919 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:23.919 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:54:23.919 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691902 2024-06-03T06:54:23.919 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691902 2024-06-03T06:54:23.919 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673065 2-64424509539 3-515396075522 4-107374182482 5-515396075522 6-150323855432 7-171798691902' 2024-06-03T06:54:23.919 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:24.138 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qclone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:54:24.138 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509540 2024-06-03T06:54:24.138 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509540 2024-06-03T06:54:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673066 2-64424509540' 2024-06-03T06:54:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T06:54:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075523 2024-06-03T06:54:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075523 2024-06-03T06:54:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673066 2-64424509540 3-515396075523' 2024-06-03T06:54:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-03T06:54:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182483 2024-06-03T06:54:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182483 2024-06-03T06:54:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673066 2-64424509540 3-515396075523 4-107374182483' 2024-06-03T06:54:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-03T06:54:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075523 2024-06-03T06:54:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075523 2024-06-03T06:54:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673066 2-64424509540 3-515396075523 4-107374182483 5-515396075523' 2024-06-03T06:54:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-03T06:54:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855433 2024-06-03T06:54:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855433 2024-06-03T06:54:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673066 2-64424509540 3-515396075523 4-107374182483 5-515396075523 6-150323855433' 2024-06-03T06:54:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-03T06:54:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691903 2024-06-03T06:54:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691903 2024-06-03T06:54:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673066 2-64424509540 3-515396075523 4-107374182483 5-515396075523 6-150323855433 7-171798691903' 2024-06-03T06:54:24.141 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:24.141 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:54:26.055 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuna/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-03T06:54:26.055 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528381 2024-06-03T06:54:26.055 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528381 2024-06-03T06:54:26.055 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673065 2-64424509539 3-515396075522 4-107374182482 5-515396075522 6-150323855432 7-171798691902 8-193273528381' 2024-06-03T06:54:26.055 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:26.055 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:54:26.056 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598797 2024-06-03T06:54:26.056 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598797 2024-06-03T06:54:26.056 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673065 2-64424509539 3-515396075522 4-107374182482 5-515396075522 6-150323855432 7-171798691902 8-193273528381 9-446676598797' 2024-06-03T06:54:26.056 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:26.056 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836592 2024-06-03T06:54:26.056 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:26.056 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:54:26.056 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836592 2024-06-03T06:54:26.056 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:26.056 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836592 2024-06-03T06:54:26.056 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2024-06-03T06:54:26.056 INFO:tasks.workunit.client.0.smithi039.stderr:278906: waiting osd.0 seq 21474836592 2024-06-03T06:54:26.056 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:54:26.057 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836592 -lt 21474836592 2024-06-03T06:54:26.057 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:26.057 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673065 2024-06-03T06:54:26.057 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:26.057 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:54:26.057 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673065 2024-06-03T06:54:26.057 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:26.057 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673065 2024-06-03T06:54:26.057 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673065' 2024-06-03T06:54:26.057 INFO:tasks.workunit.client.0.smithi039.stderr:278906: waiting osd.1 seq 42949673065 2024-06-03T06:54:26.057 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:54:26.057 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673064 -lt 42949673065 2024-06-03T06:54:26.057 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:54:26.058 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:54:26.058 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:54:26.164 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/ctu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528382 2024-06-03T06:54:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528382 2024-06-03T06:54:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673066 2-64424509540 3-515396075523 4-107374182483 5-515396075523 6-150323855433 7-171798691903 8-193273528382' 2024-06-03T06:54:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:54:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-03T06:54:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598798 2024-06-03T06:54:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598798 2024-06-03T06:54:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673066 2-64424509540 3-515396075523 4-107374182483 5-515396075523 6-150323855433 7-171798691903 8-193273528382 9-446676598798' 2024-06-03T06:54:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836593 2024-06-03T06:54:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:54:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836593 2024-06-03T06:54:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:26.166 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836593 2024-06-03T06:54:26.166 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2024-06-03T06:54:26.166 INFO:tasks.workunit.client.0.smithi039.stderr:278908: waiting osd.0 seq 21474836593 2024-06-03T06:54:26.166 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:54:26.166 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836592 -lt 21474836593 2024-06-03T06:54:26.166 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:54:26.166 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:54:26.166 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:54:26.166 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836594 -lt 21474836593 2024-06-03T06:54:26.166 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:26.166 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673066 2024-06-03T06:54:26.166 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:26.167 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:54:26.167 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673066 2024-06-03T06:54:26.167 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:26.167 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673066 2024-06-03T06:54:26.167 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673066' 2024-06-03T06:54:26.167 INFO:tasks.workunit.client.0.smithi039.stderr:278908: waiting osd.1 seq 42949673066 2024-06-03T06:54:26.167 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:54:27.037 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673066 -lt 4294ephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673066 -lt 42949673065 2024-06-03T06:54:27.037 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509539 2024-06-03T06:54:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:54:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509539 2024-06-03T06:54:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509539 2024-06-03T06:54:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509539' 2024-06-03T06:54:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:278906: waiting osd.2 seq 64424509539 2024-06-03T06:54:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:54:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509540 -lt 64424509539 2024-06-03T06:54:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-515396075522 2024-06-03T06:54:27.039 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:27.039 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:54:27.039 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-515396075522 2024-06-03T06:54:27.039 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:27.039 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075522 2024-06-03T06:54:27.039 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 515396075522' 2024-06-03T06:54:27.039 INFO:tasks.workunit.client.0.smithi039.stderr:278906: waiting osd.3 seq 515396075522 2024-06-03T06:54:27.039 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:54:27.039 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075523 -lt 515396075522 2024-06-03T06:54:27.039 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182482 2024-06-03T06:54:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:54:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182482 2024-06-03T06:54:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182482 2024-06-03T06:54:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182482' 2024-06-03T06:54:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:278906: waiting osd.4 seq 107374182482 2024-06-03T06:54:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:54:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182483 -lt 107374182482 2024-06-03T06:54:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-515396075522 2024-06-03T06:54:27.214 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:229673066 2024-06-03T06:54:27.214 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:27.214 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509540 2024-06-03T06:54:27.214 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:27.214 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:54:27.214 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509540 2024-06-03T06:54:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509540 2024-06-03T06:54:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509540' 2024-06-03T06:54:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:278908: waiting osd.2 seq 64424509540 2024-06-03T06:54:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:54:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509540 -lt 64424509540 2024-06-03T06:54:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-515396075523 2024-06-03T06:54:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T06:54:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-515396075523 2024-06-03T06:54:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075523 2024-06-03T06:54:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 515396075523' 2024-06-03T06:54:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:278908: waiting osd.3 seq 515396075523 2024-06-03T06:54:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T06:54:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075523 -lt 515396075523 2024-06-03T06:54:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182483 2024-06-03T06:54:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-03T06:54:27.216 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182483 2024-06-03T06:54:27.217 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:27.217 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182483 2024-06-03T06:54:27.217 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182483' 2024-06-03T06:54:27.217 INFO:tasks.workunit.client.0.smithi039.stderr:278908: waiting osd.4 seq 107374182483 2024-06-03T06:54:27.217 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-03T06:54:27.217 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182484 -lt 107374182483 2024-06-03T06:54:27.217 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:27.217 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-515396075523 2024-06-03T06:54:27.217 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:27.965 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-he11: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:27.966 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:54:27.966 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-515396075522 2024-06-03T06:54:27.966 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:27.966 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075522 2024-06-03T06:54:27.966 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 515396075522' 2024-06-03T06:54:27.966 INFO:tasks.workunit.client.0.smithi039.stderr:278906: waiting osd.5 seq 515396075522 2024-06-03T06:54:27.966 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:54:27.966 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075524 -lt 515396075522 2024-06-03T06:54:27.966 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:27.966 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855432 2024-06-03T06:54:27.966 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:27.966 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:54:27.966 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855432 2024-06-03T06:54:27.967 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:27.967 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855432 2024-06-03T06:54:27.967 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855432' 2024-06-03T06:54:27.967 INFO:tasks.workunit.client.0.smithi039.stderr:278906: waiting osd.6 seq 150323855432 2024-06-03T06:54:27.967 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:54:27.967 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855434 -lt 150323855432 2024-06-03T06:54:27.967 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:27.967 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691902 2024-06-03T06:54:27.967 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:27.967 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:54:27.967 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691902 2024-06-03T06:54:27.967 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:27.968 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691902 2024-06-03T06:54:27.968 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691902' 2024-06-03T06:54:27.968 INFO:tasks.workunit.client.0.smithi039.stderr:278906: waiting osd.7 seq 171798691902 2024-06-03T06:54:27.968 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:54:27.968 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691904 -lt 171798691902 2024-06-03T06:54:27.968 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:27.968 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528381 2024-06-03T06:54:27.968 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:27.968 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:54:27.968 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528381 2024-06-03T06:54:27.968 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:28.160 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.clienlpers.sh:2211: flush_pg_stats: osd=5 2024-06-03T06:54:28.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-515396075523 2024-06-03T06:54:28.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:28.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075523 2024-06-03T06:54:28.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 515396075523' 2024-06-03T06:54:28.160 INFO:tasks.workunit.client.0.smithi039.stderr:278908: waiting osd.5 seq 515396075523 2024-06-03T06:54:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-03T06:54:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075524 -lt 515396075523 2024-06-03T06:54:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855433 2024-06-03T06:54:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-03T06:54:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855433 2024-06-03T06:54:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855433 2024-06-03T06:54:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855433' 2024-06-03T06:54:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: waiting osd.6 seq 150323855433 2024-06-03T06:54:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-03T06:54:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855434 -lt 150323855433 2024-06-03T06:54:28.162 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:28.162 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691903 2024-06-03T06:54:28.162 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:28.162 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-03T06:54:28.162 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691903 2024-06-03T06:54:28.162 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:28.162 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691903 2024-06-03T06:54:28.162 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691903' 2024-06-03T06:54:28.162 INFO:tasks.workunit.client.0.smithi039.stderr:278908: waiting osd.7 seq 171798691903 2024-06-03T06:54:28.162 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-03T06:54:28.162 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691904 -lt 171798691903 2024-06-03T06:54:28.162 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:28.162 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528382 2024-06-03T06:54:28.163 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:28.163 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-03T06:54:28.163 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528382 2024-06-03T06:54:28.163 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:28.163 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528382 2024-06-03T06:54:29.659 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtet.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528381 2024-06-03T06:54:29.659 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528381' 2024-06-03T06:54:29.659 INFO:tasks.workunit.client.0.smithi039.stderr:278906: waiting osd.8 seq 193273528381 2024-06-03T06:54:29.659 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:54:29.659 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528383 -lt 193273528381 2024-06-03T06:54:29.659 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:29.659 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-446676598797 2024-06-03T06:54:29.659 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:29.659 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:54:29.659 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-446676598797 2024-06-03T06:54:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598797 2024-06-03T06:54:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 446676598797' 2024-06-03T06:54:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:278906: waiting osd.9 seq 446676598797 2024-06-03T06:54:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:54:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598799 -lt 446676598797 2024-06-03T06:54:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:54:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:54:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:54:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:54:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:54:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:54:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:54:29.661 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:54:29.661 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:54:29.661 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:54:29.661 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:54:29.661 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:54:29.661 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:54:29.661 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:54:29.661 INFO:tasks.workunit.client.0.smithi039.stderr:278906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:54:29.661 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:54:29.661 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:54:29.661 INFO:tasks.workunit.client.0.smithi039.stderr:278906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:54:29.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:54:29.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T06:54:29.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 278905 2024-06-03T06:54:29.879 INFO:tasks.workunit.client.0.smithi039.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528382' 2024-06-03T06:54:29.879 INFO:tasks.workunit.client.0.smithi039.stderr:278908: waiting osd.8 seq 193273528382 2024-06-03T06:54:29.879 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-03T06:54:29.879 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528383 -lt 193273528382 2024-06-03T06:54:29.879 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:54:29.879 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-446676598798 2024-06-03T06:54:29.879 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:54:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-03T06:54:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-446676598798 2024-06-03T06:54:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:54:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598798 2024-06-03T06:54:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 446676598798' 2024-06-03T06:54:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:278908: waiting osd.9 seq 446676598798 2024-06-03T06:54:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-03T06:54:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598799 -lt 446676598798 2024-06-03T06:54:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:54:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:54:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:54:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T06:54:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:54:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:54:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:54:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:54:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:54:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:54:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:54:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:54:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:54:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:54:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:278908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:54:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T06:54:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:54:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:278908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:54:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T06:54:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T06:54:29.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T06:54:29.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T06:54:29.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-03T06:54:29.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-03T06:54:29.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:54:29.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:54:29.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T06:54:29.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:54:29.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:54:29.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:54:29.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:54:29.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:54:29.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:54:29.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:54:29.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:54:30.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:54:30.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:54:30.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:54:30.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:54:30.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:54:30.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:54:30.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:54:30.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:54:30.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:54:30.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:54:30.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:54:30.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:54:30.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:54:30.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:54:30.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:54:30.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:54:30.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:54:30.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:54:30.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:54:30.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:54:30.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T06:54:30.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T06:54:30.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T06:54:30.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T06:54:30.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:54:30.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:54:30.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:54:30.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:54:30.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:54:30.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:54:30.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:54:30.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:54:30.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:54:30.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:54:30.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:54:30.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:54:30.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:54:30.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:54:30.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:54:30.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:54:30.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:54:30.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:54:30.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:54:30.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:54:30.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:54:30.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:54:30.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:54:30.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:54:30.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:54:30.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:54:30.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:54:30.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:54:30.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T06:54:30.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:54:30.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:54:30.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:54:30.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T06:54:30.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:54:30.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T06:54:30.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:54:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T06:54:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-06-03T06:54:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-06-03T06:54:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-06-03T06:54:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-06-03T06:54:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T06:54:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T06:54:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T06:54:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T06:54:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T06:54:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-06-03T06:54:30.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T06:54:30.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:54:30.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:54:30.504 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:54:30.504 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:54:30.504 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:54:30.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:54:30.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-03T06:54:30.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T06:54:30.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T06:54:30.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:54:30.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:54:30.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T06:54:30.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T06:54:30.707 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:54:30.707 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:54:30.707 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:54:30.707 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:54:30.707 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:54:30.707 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:54:30.707 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:54:30.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:54:30.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T06:54:30.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T06:54:30.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:54:30.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:54:30.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T06:54:30.796 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:54:30.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T06:54:30.796 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:54:30.796 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:54:30.796 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:54:30.796 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:54:30.796 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:54:30.796 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:54:30.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:54:30.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T06:54:30.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-06-03T06:54:30.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T06:54:30.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T06:54:30.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T06:54:30.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T06:54:30.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T06:54:30.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T06:54:31.698 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T06:54:31.699 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:54:31.699 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:54:31.699 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:54:31.699 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:54:31.699 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:54:31.699 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:54:31.699 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:54:31.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:54:31.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-06-03T06:54:31.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:54:31.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:54:31.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T06:54:31.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:54:31.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:54:31.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:54:31.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:54:31.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:54:31.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:54:31.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:54:31.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:54:31.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:54:31.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:54:31.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:54:31.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:54:31.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:54:31.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:54:31.732 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:54:31.732 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:54:31.732 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:54:31.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:54:31.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:54:31.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:54:31.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:54:31.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:54:31.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:54:31.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:54:31.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:54:31.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:54:31.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:54:31.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:54:31.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:54:31.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:54:31.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:54:31.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:54:31.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=15d7bb70-7157-4f37-96d3-57a761f269bf 2024-06-03T06:54:31.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 15d7bb70-7157-4f37-96d3-57a761f269bf' 2024-06-03T06:54:31.734 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 15d7bb70-7157-4f37-96d3-57a761f269bf 2024-06-03T06:54:31.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:54:31.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnaF1mgpjHLBAAyVIK3/+VQjEE1ixwzvhnAA== 2024-06-03T06:54:31.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnaF1mgpjHLBAAyVIK3/+VQjEE1ixwzvhnAA=="}' 2024-06-03T06:54:31.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 15d7bb70-7157-4f37-96d3-57a761f269bf -i td/osd-scrub-repair/0/new.json 2024-06-03T06:54:31.963 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:54:31.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T06:54:31.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCnaF1mgpjHLBAAyVIK3/+VQjEE1ixwzvhnAA== --osd-uuid 15d7bb70-7157-4f37-96d3-57a761f269bf 2024-06-03T06:54:32.004 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:54:32.003+0000 7fd6d5225640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:32.014 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:54:32.013+0000 7fd6d5225640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:32.015 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:54:32.014+0000 7fd6d5225640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:32.015 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:54:32.014+0000 7fd6d5225640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:54:32.015 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:54:32.014+0000 7fd6d5225640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:54:32.015 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:54:32.014+0000 7fd6d5225640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:54:32.106 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:54:32.105+0000 7fd6d5225640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T06:54:37.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T06:54:37.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:54:37.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T06:54:37.927 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:54:37.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:54:38.308 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:54:38.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T06:54:38.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:54:38.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:54:38.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:54:38.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:54:38.336 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:54:38.336+0000 7f5ff7818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:38.338 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:54:38.338+0000 7f5ff7818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:38.339 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:54:38.339+0000 7f5ff7818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:54:38.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T06:54:38.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:54:38.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:54:38.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:54:38.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:54:38.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:38.599 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:54:38.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:54:38.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:38.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:38.859 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:54:38.859+0000 7f5ff7818640 -1 Falling back to public interface 2024-06-03T06:54:38.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:39.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:39.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:39.887 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:54:39.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:54:39.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:39.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:40.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:41.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:41.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:41.191 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:54:41.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:54:41.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:41.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:41.299 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:54:41.299+0000 7f5ff7818640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:54:41.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:42.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:42.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:42.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:54:42.496 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:54:42.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:42.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:42.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:43.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:43.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:43.796 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:54:43.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:54:43.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:43.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:44.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:45.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:45.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:45.101 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:54:45.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:54:45.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:45.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:45.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:46.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:46.404 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:54:46.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:46.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:54:46.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:46.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:46.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:47.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:47.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:47.711 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:54:47.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:54:47.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:47.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:48.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:49.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:49.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:49.104 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:54:49.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:54:49.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:49.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:49.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:50.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:50.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:50.405 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:54:50.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:54:50.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:50.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:50.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:51.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:51.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:51.708 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:54:51.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:54:51.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:51.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:52.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:53.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:53.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:53.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:54:53.106 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:54:53.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:53.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:53.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:54.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:54.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:54.608 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:54:54.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:54:54.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:54.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:55.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:56.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:56.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:56.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:54:56.227 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:54:56.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:56.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:56.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:57.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:57.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:57.887 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:54:57.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:54:57.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:57.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:58.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:54:59.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:54:59.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:54:59.264 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:54:59.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:54:59.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:54:59.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:54:59.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:00.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:00.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:00.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:55:00.863 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:55:00.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:00.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:55:01.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:02.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:02.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:02.532 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:55:02.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:55:02.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:02.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:55:03.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:04.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:04.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:04.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:55:04.242 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:55:04.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:04.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:55:04.540 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1748650836,v1:127.0.0.1:6803/1748650836] [v2:127.0.0.1:6804/1748650836,v1:127.0.0.1:6805/1748650836] exists,up 15d7bb70-7157-4f37-96d3-57a761f269bf 2024-06-03T06:55:04.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:55:04.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:55:04.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:55:04.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-06-03T06:55:04.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:55:04.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:55:04.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T06:55:04.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:55:04.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:55:04.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:55:04.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:55:04.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:55:04.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:55:04.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:55:04.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:55:04.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:55:04.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:55:04.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:55:04.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:55:04.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:55:04.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:55:04.546 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:55:04.546 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:55:04.546 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:55:04.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:55:04.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:55:04.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:55:04.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:55:04.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:55:04.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:55:04.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:55:04.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:55:04.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:55:04.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:55:04.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:55:04.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:55:04.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:55:04.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:55:04.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:55:04.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7bf1ad8e-9dd9-4e27-8d5c-175a8eb00fe3 2024-06-03T06:55:04.548 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 7bf1ad8e-9dd9-4e27-8d5c-175a8eb00fe3 2024-06-03T06:55:04.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7bf1ad8e-9dd9-4e27-8d5c-175a8eb00fe3' 2024-06-03T06:55:04.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:55:04.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDIaF1mtuuKIRAAD+gZ0W1iwxmy/ymi4EeoIQ== 2024-06-03T06:55:04.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDIaF1mtuuKIRAAD+gZ0W1iwxmy/ymi4EeoIQ=="}' 2024-06-03T06:55:04.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7bf1ad8e-9dd9-4e27-8d5c-175a8eb00fe3 -i td/osd-scrub-repair/1/new.json 2024-06-03T06:55:04.900 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:55:04.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T06:55:04.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDIaF1mtuuKIRAAD+gZ0W1iwxmy/ymi4EeoIQ== --osd-uuid 7bf1ad8e-9dd9-4e27-8d5c-175a8eb00fe3 2024-06-03T06:55:04.941 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:04.941+0000 7f5939744640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:55:04.993 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:04.993+0000 7f5939744640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:55:04.994 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:04.994+0000 7f5939744640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:55:04.995 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:04.995+0000 7f5939744640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:55:04.995 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:04.995+0000 7f5939744640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:55:04.995 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:04.995+0000 7f5939744640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:55:05.068 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:05.068+0000 7f5939744640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T06:55:11.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T06:55:11.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:55:11.305 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:55:11.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T06:55:11.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:55:11.681 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:55:11.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T06:55:11.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:55:11.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:55:11.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:55:11.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:55:11.708 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:11.708+0000 7f287ae2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:55:11.710 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:11.711+0000 7f287ae2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:55:11.711 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:11.712+0000 7f287ae2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:55:11.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T06:55:11.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:55:11.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:55:11.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:55:11.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:55:11.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:11.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:55:11.974 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:55:11.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:11.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:12.230 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:12.230+0000 7f287ae2c640 -1 Falling back to public interface 2024-06-03T06:55:12.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:13.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:13.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:13.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:55:13.282 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:55:13.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:13.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:13.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:14.570 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:14.569+0000 7f287ae2c640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:55:14.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:14.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:14.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:55:14.582 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:55:14.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:14.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:14.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:15.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:15.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:15.879 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:55:15.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:55:15.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:15.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:16.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:17.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:17.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:17.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:55:17.218 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:55:17.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:17.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:17.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:18.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:18.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:18.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:55:18.520 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:55:18.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:18.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:18.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:19.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:19.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:19.879 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:55:19.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:55:19.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:19.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:20.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:21.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:21.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:21.218 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:55:21.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:55:21.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:21.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:21.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:22.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:22.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:22.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:55:22.626 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:55:22.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:22.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:23.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:24.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:24.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:24.065 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:55:24.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:55:24.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:24.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:24.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:25.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:25.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:25.454 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:55:25.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:55:25.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:25.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:25.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:26.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:26.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:26.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:55:26.758 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:55:26.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:26.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:27.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:28.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:28.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:28.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:55:28.098 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:55:28.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:28.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:28.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:29.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:29.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:29.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:55:29.577 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:55:29.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:29.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:30.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:31.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:31.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:31.029 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:55:31.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:55:31.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:31.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:31.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:32.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:32.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:32.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:55:32.553 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:55:32.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:32.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:32.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:33.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:33.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:33.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:55:33.859 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:55:33.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:33.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:34.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:35.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:35.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:35.319 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:55:35.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:55:35.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:35.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:35.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:36.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:36.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:36.834 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:55:36.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:55:36.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:36.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:37.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:38.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:38.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:38.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:55:38.493 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:55:38.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:38.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:55:38.819 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1622731230,v1:127.0.0.1:6811/1622731230] [v2:127.0.0.1:6812/1622731230,v1:127.0.0.1:6813/1622731230] exists,up 7bf1ad8e-9dd9-4e27-8d5c-175a8eb00fe3 2024-06-03T06:55:38.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:55:38.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:55:38.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:55:38.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-06-03T06:55:38.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T06:55:39.101 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-06-03T06:55:39.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T06:55:39.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T06:55:39.509 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T06:55:39.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:55:40.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T06:55:42.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-06-03T06:55:42.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:55:42.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:55:42.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:55:42.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:55:42.140 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:55:42.140 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:55:42.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:55:42.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:55:42.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:55:42.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:55:42.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:55:42.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:55:42.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:55:42.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:55:42.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:55:42.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:55:42.553 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T06:55:42.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:55:42.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:55:42.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:55:42.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836494 2024-06-03T06:55:42.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836494 2024-06-03T06:55:42.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494' 2024-06-03T06:55:42.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:55:42.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:55:42.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-06-03T06:55:42.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-06-03T06:55:42.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-42949672966' 2024-06-03T06:55:42.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:55:42.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836494 2024-06-03T06:55:42.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:55:42.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:55:42.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836494 2024-06-03T06:55:42.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:55:42.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836494 2024-06-03T06:55:42.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-06-03T06:55:42.777 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836494 2024-06-03T06:55:42.777 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:55:43.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2024-06-03T06:55:43.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:55:44.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:55:44.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:55:44.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836494 2024-06-03T06:55:44.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:55:44.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-06-03T06:55:44.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:55:44.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:55:44.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-06-03T06:55:44.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:55:44.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-06-03T06:55:44.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-06-03T06:55:44.373 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672966 2024-06-03T06:55:44.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:55:44.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672966 2024-06-03T06:55:44.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:55:44.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:55:44.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:55:45.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T06:55:45.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:55:45.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:55:45.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:55:45.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:55:45.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:55:45.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:55:45.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:55:45.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:55:45.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:55:45.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:55:45.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:55:45.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T06:55:45.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:55:45.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:55:45.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-06-03T06:55:45.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T06:55:45.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-06-03T06:55:45.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-03T06:55:45.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T06:55:45.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T06:55:45.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T06:55:46.076 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T06:55:46.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T06:55:46.654 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T06:55:46.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T06:55:46.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T06:55:46.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-03T06:55:46.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-06-03T06:55:46.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-06-03T06:55:46.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-03T06:55:46.842 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-06-03T06:55:46.842 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-06-03T06:55:46.842 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:55:46.842 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-06-03T06:55:46.842 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:55:47.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T06:55:47.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-06-03T06:55:47.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T06:55:47.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-06-03T06:55:47.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-03T06:55:47.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-06-03T06:55:47.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-06-03T06:55:47.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-06-03T06:55:47.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:55:47.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:55:47.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:55:47.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:55:47.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-06-03T06:55:47.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:55:47.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:55:47.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:55:47.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:55:47.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:55:47.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:55:47.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:55:47.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:55:47.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:55:47.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:55:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:55:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-06-03T06:55:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:55:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:55:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:55:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:55:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:55:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-06-03T06:55:49.286 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-03T06:55:49.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T06:55:49.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:55:49.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:55:49.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:55:49.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:55:49.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:55:49.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:55:49.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:55:49.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:55:49.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:55:49.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:55:49.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:55:49.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:55:49.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:55:49.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:55:49.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:55:49.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:55:49.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:55:49.802 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:55:49.802 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:55:49.802 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:55:49.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:55:49.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:55:49.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:55:49.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:55:49.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:55:49.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:55:49.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:55:49.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:55:49.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:55:49.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:55:49.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:55:49.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:55:49.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:55:49.804 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:55:49.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:55:49.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:55:49.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:55:49.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:55:49.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:55:49.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:55:49.834 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:49.834+0000 7f98f01a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:55:49.836 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:49.836+0000 7f98f01a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:55:49.837 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:49.837+0000 7f98f01a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:55:50.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:55:50.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:55:50.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:55:50.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:55:50.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:55:50.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:50.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:55:50.104 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:55:50.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:50.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:55:50.356 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:50.356+0000 7f98f01a3640 -1 Falling back to public interface 2024-06-03T06:55:50.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:51.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:51.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:51.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:55:51.392 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:55:51.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:51.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:55:51.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:52.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:52.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:52.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:55:52.693 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:55:52.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:52.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:55:52.716 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:55:52.716+0000 7f98f01a3640 -1 osd.0 20 log_to_monitors true 2024-06-03T06:55:53.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:55:54.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:55:54.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:55:54.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:55:54.082 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:55:54.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:55:54.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:55:54.427 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 24 up_thru 24 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/3847300468,v1:127.0.0.1:6803/3847300468] [v2:127.0.0.1:6804/3847300468,v1:127.0.0.1:6805/3847300468] exists,up 15d7bb70-7157-4f37-96d3-57a761f269bf 2024-06-03T06:55:54.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:55:54.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:55:54.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:55:54.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:55:54.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:55:54.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:55:54.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:55:54.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:55:54.428 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:55:54.428 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:55:54.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:55:54.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:55:54.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:55:54.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:55:54.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:55:54.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:55:54.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:55:54.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:55:54.522 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:55:54.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:55:54.836 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T06:55:54.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:55:54.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:55:54.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:55:54.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215106 2024-06-03T06:55:54.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215106 2024-06-03T06:55:54.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106' 2024-06-03T06:55:54.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:55:54.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:55:55.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-06-03T06:55:55.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-06-03T06:55:55.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672970' 2024-06-03T06:55:55.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:55:55.073 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215106 2024-06-03T06:55:55.073 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:55:55.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:55:55.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215106 2024-06-03T06:55:55.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:55:55.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215106 2024-06-03T06:55:55.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-06-03T06:55:55.075 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 103079215106 2024-06-03T06:55:55.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:55:55.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 103079215106 2024-06-03T06:55:55.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:55:56.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:55:56.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:55:56.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215106 -lt 103079215106 2024-06-03T06:55:56.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:55:56.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-06-03T06:55:56.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:55:56.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:55:56.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-06-03T06:55:56.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:55:56.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-06-03T06:55:56.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-06-03T06:55:56.669 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672970 2024-06-03T06:55:56.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:55:57.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-06-03T06:55:57.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:55:57.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:55:57.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:55:57.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T06:55:57.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:55:57.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:55:57.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:55:57.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:55:57.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:55:57.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:55:57.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:55:57.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:55:57.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:55:57.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:55:57.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:55:58.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T06:55:58.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:55:58.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:55:58.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-06-03T06:55:58.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-06-03T06:55:58.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:55:58.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-06-03T06:55:58.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:55:58.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-06-03T06:55:58.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-06-03T06:55:58.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-06-03T06:55:58.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-06-03T06:55:58.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:55:58.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:55:58.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:55:58.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:55:58.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:55:39.453724+0000 2024-06-03T06:55:58.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-06-03T06:55:58.933 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 1.3 on osd.1 to repair 2024-06-03T06:55:58.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-06-03T06:55:39.453724+0000 2024-06-03T06:55:58.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-06-03T06:55:58.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:55:39.453724+0000 2024-06-03T06:55:58.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:55:58.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:55:58.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:55:58.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:55:58.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:55:58.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:55:58.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:55:58.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:55:59.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:55:39.453724+0000 '>' 2024-06-03T06:55:39.453724+0000 2024-06-03T06:55:59.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:56:00.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:56:00.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:56:00.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:56:00.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:56:00.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:56:00.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:56:00.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:56:00.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:55:39.453724+0000 '>' 2024-06-03T06:55:39.453724+0000 2024-06-03T06:56:00.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:56:01.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:56:01.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:56:01.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:56:01.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:56:01.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:56:01.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:56:01.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:56:01.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:55:39.453724+0000 '>' 2024-06-03T06:55:39.453724+0000 2024-06-03T06:56:01.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:56:02.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:56:02.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:56:02.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:56:02.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:56:02.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:56:02.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:56:02.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:56:03.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:55:39.453724+0000 '>' 2024-06-03T06:55:39.453724+0000 2024-06-03T06:56:03.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:56:04.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:56:04.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:56:04.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:56:04.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:56:04.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:56:04.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:56:04.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:56:04.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:55:39.453724+0000 '>' 2024-06-03T06:55:39.453724+0000 2024-06-03T06:56:04.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:56:05.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:56:05.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:56:05.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:56:05.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:56:05.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:56:05.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:56:05.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:56:05.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:55:39.453724+0000 '>' 2024-06-03T06:55:39.453724+0000 2024-06-03T06:56:05.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:56:06.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:56:06.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:56:06.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:56:06.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:56:06.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:56:06.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:56:06.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:56:07.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:55:59.493744+0000 '>' 2024-06-03T06:55:39.453724+0000 2024-06-03T06:56:07.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:56:07.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:56:07.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:56:07.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:56:07.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:56:07.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:56:07.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:56:07.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:56:07.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:56:07.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:56:07.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:56:07.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:56:07.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:56:07.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:56:07.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:56:07.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:56:07.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:56:07.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:56:07.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:56:07.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:56:07.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:56:07.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:56:07.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:56:07.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:56:07.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-03T06:56:08.695 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-03T06:56:08.695 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:56:08.696 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:56:08.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:56:08.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:56:08.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:56:08.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:56:08.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:56:08.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:56:08.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:56:08.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:56:08.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:56:08.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:56:08.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:56:08.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:56:08.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:56:08.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:56:08.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:56:08.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:56:08.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:56:08.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:56:08.969 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:56:08.969 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:56:08.969 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:56:08.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:56:08.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:56:08.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:56:08.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:56:08.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:56:08.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:56:08.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:56:08.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:56:08.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:56:08.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:56:08.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:56:08.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:56:08.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:56:08.970 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:56:08.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:56:08.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:56:08.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:56:08.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:56:08.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:56:08.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:56:08.999 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:08.998+0000 7f8a46c2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:56:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:09.000+0000 7f8a46c2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:56:09.002 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:09.002+0000 7f8a46c2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:56:09.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:56:09.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:56:09.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:56:09.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:56:09.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:56:09.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:09.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:56:09.276 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:56:09.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:09.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:56:09.520 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:09.519+0000 7f8a46c2c640 -1 Falling back to public interface 2024-06-03T06:56:09.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:56:10.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:56:10.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:10.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:56:10.564 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:56:10.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:10.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:56:10.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:56:11.750 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:11.749+0000 7f8a46c2c640 -1 osd.1 26 log_to_monitors true 2024-06-03T06:56:11.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:56:11.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:11.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:56:11.898 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:56:11.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:11.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:56:12.408 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 30 up_thru 25 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/1424680432,v1:127.0.0.1:6811/1424680432] [v2:127.0.0.1:6812/1424680432,v1:127.0.0.1:6813/1424680432] exists,up 7bf1ad8e-9dd9-4e27-8d5c-175a8eb00fe3 2024-06-03T06:56:12.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:56:12.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:56:12.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:56:12.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:56:12.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:56:12.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:56:12.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:56:12.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:56:12.409 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:56:12.409 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:56:12.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:56:12.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:56:12.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:56:12.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:56:12.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:56:12.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:56:12.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:56:12.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:56:12.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:56:12.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:56:12.795 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T06:56:12.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:56:12.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:56:12.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:56:12.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215110 2024-06-03T06:56:12.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215110 2024-06-03T06:56:12.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110' 2024-06-03T06:56:12.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:56:12.904 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:56:13.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-06-03T06:56:13.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-06-03T06:56:13.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110 1-128849018882' 2024-06-03T06:56:13.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:56:13.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215110 2024-06-03T06:56:13.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:56:13.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:56:13.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215110 2024-06-03T06:56:13.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:56:13.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215110 2024-06-03T06:56:13.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215110' 2024-06-03T06:56:13.039 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 103079215110 2024-06-03T06:56:13.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:56:13.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215109 -lt 103079215110 2024-06-03T06:56:13.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:56:14.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:56:14.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:56:14.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215111 -lt 103079215110 2024-06-03T06:56:14.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:56:14.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018882 2024-06-03T06:56:14.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:56:14.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:56:14.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018882 2024-06-03T06:56:14.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:56:14.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-06-03T06:56:14.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-06-03T06:56:14.635 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 128849018882 2024-06-03T06:56:14.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:56:14.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-06-03T06:56:14.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:56:14.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:56:14.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:56:15.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T06:56:15.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:56:15.307 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:56:15.307 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:56:15.307 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:56:15.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:56:15.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:56:15.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:56:15.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:56:15.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:56:15.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:56:15.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:56:15.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T06:56:15.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:56:15.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:56:15.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:56:16.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:56:16.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-06-03T06:56:16.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-06-03T06:56:16.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-03T06:56:16.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-06-03T06:56:16.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T06:56:16.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-06-03T06:56:16.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-03T06:56:16.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-06-03T06:56:16.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-06-03T06:56:16.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:56:16.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:56:16.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:56:16.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:56:16.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:56:16.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:56:16.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:56:16.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:56:16.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:56:16.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:56:16.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:56:16.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:56:16.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:56:16.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:56:16.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:56:16.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:56:16.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:56:16.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-03T06:56:16.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:56:16.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:56:16.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:56:16.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:56:16.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:56:16.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-03T06:56:17.878 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-03T06:56:18.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:56:18.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:56:18.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:56:18.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:56:18.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:56:18.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:56:18.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:56:18.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:56:18.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:56:18.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:56:18.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:56:18.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:56:18.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:56:18.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:56:18.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:56:18.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:56:18.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:56:18.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:56:18.397 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:56:18.397 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:56:18.397 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:56:18.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:56:18.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:56:18.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:56:18.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:56:18.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:56:18.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:56:18.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:56:18.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:56:18.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:56:18.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:56:18.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:56:18.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:56:18.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:56:18.399 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:56:18.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:56:18.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:56:18.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:56:18.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:56:18.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:56:18.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:56:18.428 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:18.428+0000 7ff98a41f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:56:18.430 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:18.431+0000 7ff98a41f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:56:18.432 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:18.432+0000 7ff98a41f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:56:18.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:56:18.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:56:18.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:56:18.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:56:18.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:56:18.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:18.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:56:18.698 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:56:18.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:18.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:56:18.948 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:18.948+0000 7ff98a41f640 -1 Falling back to public interface 2024-06-03T06:56:19.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:56:20.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:56:20.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:20.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:56:20.002 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:56:20.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:20.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:56:20.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:56:21.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:56:21.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:21.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:56:21.302 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:56:21.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:21.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:56:21.327 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:21.327+0000 7ff98a41f640 -1 osd.1 31 log_to_monitors true 2024-06-03T06:56:21.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:56:22.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:56:22.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:22.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:56:22.657 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:56:22.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:22.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:56:22.964 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6810/1410836346,v1:127.0.0.1:6811/1410836346] [v2:127.0.0.1:6812/1410836346,v1:127.0.0.1:6813/1410836346] exists,up 7bf1ad8e-9dd9-4e27-8d5c-175a8eb00fe3 2024-06-03T06:56:22.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:56:22.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:56:22.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:56:22.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:56:22.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:56:22.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:56:22.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:56:22.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:56:22.966 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:56:22.966 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:56:22.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:56:22.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:56:22.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:56:23.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:56:23.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:56:23.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:56:23.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:56:23.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:56:23.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:56:23.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:56:23.347 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T06:56:23.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:56:23.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:56:23.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:56:23.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215114 2024-06-03T06:56:23.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215114 2024-06-03T06:56:23.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215114' 2024-06-03T06:56:23.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:56:23.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:56:23.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-06-03T06:56:23.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-06-03T06:56:23.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215114 1-150323855362' 2024-06-03T06:56:23.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:56:23.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215114 2024-06-03T06:56:23.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:56:23.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:56:23.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215114 2024-06-03T06:56:23.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:56:23.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215114 2024-06-03T06:56:23.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215114' 2024-06-03T06:56:23.566 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 103079215114 2024-06-03T06:56:23.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:56:23.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215112 -lt 103079215114 2024-06-03T06:56:23.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:56:24.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:56:24.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:56:25.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215114 -lt 103079215114 2024-06-03T06:56:25.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:56:25.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-06-03T06:56:25.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:56:25.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:56:25.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-06-03T06:56:25.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:56:25.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-06-03T06:56:25.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-06-03T06:56:25.204 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 150323855362 2024-06-03T06:56:25.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:56:25.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-06-03T06:56:25.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:56:25.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:56:25.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:56:25.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T06:56:25.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:56:25.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:56:25.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:56:25.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:56:25.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:56:25.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:56:25.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:56:26.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:56:26.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:56:26.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:56:26.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:56:26.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T06:56:26.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:56:26.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:56:26.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-06-03T06:56:26.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-06-03T06:56:26.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-03T06:56:26.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-06-03T06:56:26.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T06:56:26.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-06-03T06:56:26.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-06-03T06:56:26.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-06-03T06:56:26.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-06-03T06:56:26.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:56:26.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:56:26.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:56:26.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:56:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T06:55:59.493744+0000 2024-06-03T06:56:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-06-03T06:56:27.314 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 1.3 on osd.1 to repair 2024-06-03T06:56:27.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-06-03T06:55:59.493744+0000 2024-06-03T06:56:27.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-06-03T06:56:27.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:55:59.493744+0000 2024-06-03T06:56:27.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T06:56:27.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T06:56:27.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:56:27.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:56:27.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:56:27.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:56:27.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:56:27.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:56:27.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:55:59.493744+0000 '>' 2024-06-03T06:55:59.493744+0000 2024-06-03T06:56:27.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:56:28.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:56:28.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:56:28.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:56:28.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:56:28.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:56:28.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:56:28.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:56:28.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:55:59.493744+0000 '>' 2024-06-03T06:55:59.493744+0000 2024-06-03T06:56:28.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:56:29.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:56:29.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:56:29.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:56:29.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:56:29.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:56:29.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:56:29.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:56:30.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:55:59.493744+0000 '>' 2024-06-03T06:55:59.493744+0000 2024-06-03T06:56:30.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:56:31.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:56:31.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:56:31.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:56:31.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:56:31.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:56:31.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:56:31.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:56:31.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:55:59.493744+0000 '>' 2024-06-03T06:55:59.493744+0000 2024-06-03T06:56:31.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T06:56:32.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T06:56:32.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T06:56:32.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-03T06:56:32.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-03T06:56:32.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T06:56:32.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T06:56:32.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-03T06:56:32.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:56:27.415196+0000 '>' 2024-06-03T06:55:59.493744+0000 2024-06-03T06:56:32.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T06:56:32.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:56:32.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:56:32.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:56:32.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:56:32.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:56:32.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:56:32.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:56:32.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:56:32.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:56:32.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:56:32.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:56:32.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:56:32.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:56:32.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:56:32.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:56:32.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:56:33.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:56:33.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-03T06:56:33.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:56:33.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:56:33.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:56:33.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:56:33.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:56:33.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-03T06:56:34.575 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-03T06:56:34.575 INFO:tasks.workunit.client.0.smithi039.stdout:_ 2024-06-03T06:56:34.575 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 2024-06-03T06:56:34.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:56:34.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:56:34.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:56:34.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:56:34.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:56:34.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:56:34.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:56:34.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:56:34.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:56:34.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:56:34.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:56:34.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:56:34.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:56:34.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:56:34.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:56:34.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:56:34.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:56:34.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:56:34.848 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:56:34.848 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:56:34.848 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:56:34.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:56:34.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:56:34.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:56:34.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:56:34.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:56:34.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:56:34.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:56:34.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:56:34.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:56:34.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:56:34.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:56:34.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:56:34.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:56:34.850 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:56:34.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:56:34.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:56:34.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:56:34.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:56:34.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:56:34.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:56:34.878 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:34.878+0000 7fa8de62e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:56:34.912 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:34.913+0000 7fa8de62e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:56:34.914 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:34.914+0000 7fa8de62e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:56:35.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:56:35.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:56:35.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:56:35.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:56:35.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:56:35.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:35.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:56:35.153 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:56:35.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:35.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:56:35.428 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:35.428+0000 7fa8de62e640 -1 Falling back to public interface 2024-06-03T06:56:35.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:56:36.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:56:36.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:36.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:56:36.444 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:56:36.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:36.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:56:36.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:56:37.733 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:37.733+0000 7fa8de62e640 -1 osd.1 37 log_to_monitors true 2024-06-03T06:56:37.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:56:37.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:37.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:56:37.744 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:56:37.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:37.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:56:38.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:56:39.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:56:39.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:39.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:56:39.049 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:56:39.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:39.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:56:39.377 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [35,37) [v2:127.0.0.1:6810/1521722762,v1:127.0.0.1:6811/1521722762] [v2:127.0.0.1:6812/1521722762,v1:127.0.0.1:6813/1521722762] exists,up 7bf1ad8e-9dd9-4e27-8d5c-175a8eb00fe3 2024-06-03T06:56:39.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:56:39.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:56:39.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:56:39.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:56:39.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:56:39.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:56:39.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:56:39.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:56:39.378 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:56:39.378 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:56:39.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:56:39.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:56:39.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:56:39.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:56:39.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:56:39.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:56:39.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:56:39.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:56:39.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:56:39.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:56:39.758 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T06:56:39.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:56:39.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:56:39.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:56:39.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215118 2024-06-03T06:56:39.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215118 2024-06-03T06:56:39.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215118' 2024-06-03T06:56:39.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:56:39.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:56:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659138 2024-06-03T06:56:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659138 2024-06-03T06:56:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215118 1-176093659138' 2024-06-03T06:56:39.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:56:39.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215118 2024-06-03T06:56:39.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:56:39.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:56:39.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215118 2024-06-03T06:56:39.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:56:39.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215118 2024-06-03T06:56:39.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215118' 2024-06-03T06:56:39.977 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 103079215118 2024-06-03T06:56:39.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:56:40.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215118 -lt 103079215118 2024-06-03T06:56:40.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:56:40.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-176093659138 2024-06-03T06:56:40.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:56:40.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:56:40.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-176093659138 2024-06-03T06:56:40.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:56:40.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659138 2024-06-03T06:56:40.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2024-06-03T06:56:40.273 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 176093659138 2024-06-03T06:56:40.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:56:40.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 176093659138 2024-06-03T06:56:40.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:56:41.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:56:41.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:56:41.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 176093659138 2024-06-03T06:56:41.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:56:42.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T06:56:42.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:56:43.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659138 -lt 176093659138 2024-06-03T06:56:43.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:56:43.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:56:43.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:56:43.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T06:56:43.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:56:43.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:56:43.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:56:43.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:56:43.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:56:43.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:56:43.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:56:43.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T06:56:43.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:56:43.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:56:43.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:56:44.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T06:56:44.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:56:44.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:56:44.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-06-03T06:56:44.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-03T06:56:44.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T06:56:44.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:56:44.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:56:44.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:56:44.206 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:56:44.206 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:56:44.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:56:44.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:56:44.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:56:44.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:56:44.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:56:44.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:56:44.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:56:44.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:56:44.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:56:44.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:56:44.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:56:44.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:56:44.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:56:44.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:56:44.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:56:44.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:56:44.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:56:44.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:56:44.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:56:44.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:56:44.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:56:44.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:56:44.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:56:44.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T06:56:44.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T06:56:44.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T06:56:44.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T06:56:44.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T06:56:44.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T06:56:44.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T06:56:44.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:56:44.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:56:44.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:56:44.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:56:44.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:56:44.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:56:44.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T06:56:44.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T06:56:44.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T06:56:44.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T06:56:44.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T06:56:44.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T06:56:44.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:56:44.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T06:56:44.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T06:56:44.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T06:56:44.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T06:56:44.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T06:56:44.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T06:56:44.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T06:56:44.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:56:44.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:56:44.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T06:56:44.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T06:56:44.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T06:56:44.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T06:56:44.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T06:56:44.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:56:44.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:56:44.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T06:56:44.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T06:56:44.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T06:56:44.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T06:56:44.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T06:56:44.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-06-03T06:56:44.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-06-03T06:56:44.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-06-03T06:56:44.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-06-03T06:56:44.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-06-03T06:56:44.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-06-03T06:56:44.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-06-03T06:56:44.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T06:56:44.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T06:56:44.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T06:56:44.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T06:56:44.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T06:56:44.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T06:56:44.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T06:56:44.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:56:44.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:56:44.735 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:56:44.735 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:56:44.735 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:56:44.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:56:44.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T06:56:44.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T06:56:44.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T06:56:44.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:56:44.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:56:44.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T06:56:44.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T06:56:44.913 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:56:44.913 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:56:44.913 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:56:44.913 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:56:44.913 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:56:44.913 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:56:44.913 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:56:44.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:56:44.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T06:56:44.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T06:56:44.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T06:56:44.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T06:56:44.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T06:56:44.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T06:56:44.999 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T06:56:44.999 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T06:56:44.999 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T06:56:45.000 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T06:56:45.000 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:56:45.000 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:56:45.000 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T06:56:45.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T06:56:45.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T06:56:45.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-06-03T06:56:45.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T06:56:45.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T06:56:45.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T06:56:45.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T06:56:45.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T06:56:45.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T06:56:45.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T06:56:45.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:56:45.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:56:45.897 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:56:45.897 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:56:45.897 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:56:45.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:56:45.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:56:45.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T06:56:45.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-06-03T06:56:45.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-03T06:56:45.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-06-03T06:56:45.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:56:45.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:56:45.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T06:56:45.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:56:45.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:56:45.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:56:45.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:56:45.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:56:45.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:56:45.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:56:45.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:56:45.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:56:45.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:56:45.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:56:45.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:56:45.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:56:45.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:56:45.930 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:56:45.930 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:56:45.930 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:56:45.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:56:45.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:56:45.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:56:45.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:56:45.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:56:45.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:56:45.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:56:45.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:56:45.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:56:45.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:56:45.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:56:45.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:56:45.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:56:45.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:56:45.932 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:56:45.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6b7a2520-f2c7-451d-9603-1bd397e95ea5 2024-06-03T06:56:45.933 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 6b7a2520-f2c7-451d-9603-1bd397e95ea5 2024-06-03T06:56:45.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6b7a2520-f2c7-451d-9603-1bd397e95ea5' 2024-06-03T06:56:45.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:56:45.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAtaV1mrKerOBAA3BV6gDp5114AKaVb0y6yPg== 2024-06-03T06:56:45.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAtaV1mrKerOBAA3BV6gDp5114AKaVb0y6yPg=="}' 2024-06-03T06:56:45.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6b7a2520-f2c7-451d-9603-1bd397e95ea5 -i td/osd-scrub-repair/0/new.json 2024-06-03T06:56:46.161 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:56:46.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T06:56:46.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAtaV1mrKerOBAA3BV6gDp5114AKaVb0y6yPg== --osd-uuid 6b7a2520-f2c7-451d-9603-1bd397e95ea5 2024-06-03T06:56:46.201 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:46.201+0000 7fa7df415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:56:46.212 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:46.212+0000 7fa7df415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:56:46.213 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:46.213+0000 7fa7df415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:56:46.213 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:46.213+0000 7fa7df415640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:56:46.213 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:46.213+0000 7fa7df415640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:56:46.213 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:46.213+0000 7fa7df415640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T06:56:46.312 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:46.312+0000 7fa7df415640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T06:56:52.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T06:56:52.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:56:52.073 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T06:56:52.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T06:56:52.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:56:52.462 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T06:56:52.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T06:56:52.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:56:52.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:56:52.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:56:52.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:56:52.490 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:52.490+0000 7f76fc901640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:56:52.492 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:52.492+0000 7f76fc901640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:56:52.493 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:52.493+0000 7f76fc901640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:56:52.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T06:56:52.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:56:52.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:56:52.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:56:52.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:56:52.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:52.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:56:52.755 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:56:52.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:52.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:56:53.011 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:53.011+0000 7f76fc901640 -1 Falling back to public interface 2024-06-03T06:56:53.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:56:54.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:56:54.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:54.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:56:54.047 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:56:54.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:54.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:56:54.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:56:55.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:56:55.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:55.350 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:56:55.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:56:55.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:55.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:56:55.353 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:56:55.354+0000 7f76fc901640 -1 osd.0 0 log_to_monitors true 2024-06-03T06:56:55.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:56:56.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:56:56.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:56.806 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:56:56.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:56:56.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:56.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:56:57.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:56:58.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:56:58.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:58.170 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:56:58.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:56:58.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:58.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:56:58.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:56:59.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:56:59.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:56:59.476 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:56:59.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:56:59.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:56:59.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:56:59.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:00.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:00.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:00.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:57:00.782 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:57:00.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:00.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:57:01.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:02.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:02.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:02.128 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:57:02.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:57:02.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:02.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:57:02.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:03.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:03.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:03.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:57:03.432 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:57:03.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:03.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:57:03.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:04.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:04.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:04.732 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:57:04.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:57:04.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:04.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:57:05.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:06.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:06.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:06.036 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:57:06.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:57:06.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:06.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:57:06.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:07.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:07.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:07.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:57:07.344 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:57:07.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:07.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:57:07.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:08.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:08.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:08.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:57:08.777 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:57:08.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:08.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:57:09.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:10.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:10.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:10.562 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:57:10.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:57:10.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:10.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:57:11.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:12.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:12.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:12.185 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:57:12.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:57:12.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:12.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:57:12.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:13.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:13.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:13.973 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:57:13.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:57:13.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:13.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:57:14.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:15.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:15.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:15.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:57:15.698 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:57:15.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:15.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:57:16.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:17.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:17.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:17.484 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:57:17.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:57:17.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:17.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:57:17.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:18.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:18.788 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:57:18.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:18.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:57:18.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:18.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:57:19.086 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/660916277,v1:127.0.0.1:6803/660916277] [v2:127.0.0.1:6804/660916277,v1:127.0.0.1:6805/660916277] exists,up 6b7a2520-f2c7-451d-9603-1bd397e95ea5 2024-06-03T06:57:19.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:57:19.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:57:19.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:57:19.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-03T06:57:19.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-06-03T06:57:19.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:57:19.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:57:19.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T06:57:19.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:57:19.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:57:19.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:57:19.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:57:19.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:57:19.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:57:19.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:57:19.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:57:19.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:57:19.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:57:19.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:57:19.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:57:19.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:57:19.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:57:19.091 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:57:19.091 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:57:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:57:19.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:57:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:57:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:57:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:57:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:57:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:57:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:57:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:57:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:57:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:57:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:57:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:57:19.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:57:19.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:57:19.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:57:19.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=82b55f3e-34c9-4014-a1f2-409ee6945bcd 2024-06-03T06:57:19.093 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 82b55f3e-34c9-4014-a1f2-409ee6945bcd 2024-06-03T06:57:19.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 82b55f3e-34c9-4014-a1f2-409ee6945bcd' 2024-06-03T06:57:19.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:57:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBPaV1mqOGBBhAA/z2AUyFLvodgH4UQ5D2Z3g== 2024-06-03T06:57:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBPaV1mqOGBBhAA/z2AUyFLvodgH4UQ5D2Z3g=="}' 2024-06-03T06:57:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 82b55f3e-34c9-4014-a1f2-409ee6945bcd -i td/osd-scrub-repair/1/new.json 2024-06-03T06:57:19.488 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:57:19.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T06:57:19.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBPaV1mqOGBBhAA/z2AUyFLvodgH4UQ5D2Z3g== --osd-uuid 82b55f3e-34c9-4014-a1f2-409ee6945bcd 2024-06-03T06:57:19.531 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:19.531+0000 7fd87a7b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:57:19.573 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:19.573+0000 7fd87a7b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:57:19.574 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:19.574+0000 7fd87a7b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:57:19.575 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:19.575+0000 7fd87a7b4640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:57:19.575 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:19.575+0000 7fd87a7b4640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:57:19.575 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:19.575+0000 7fd87a7b4640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T06:57:19.650 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:19.650+0000 7fd87a7b4640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T06:57:25.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T06:57:25.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:57:25.992 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T06:57:25.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T06:57:25.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:57:26.397 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T06:57:26.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T06:57:26.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:57:26.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:57:26.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:57:26.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:57:26.425 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:26.425+0000 7fb0f6c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:57:26.427 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:26.428+0000 7fb0f6c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:57:26.428 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:26.429+0000 7fb0f6c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:57:26.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T06:57:26.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:57:26.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:57:26.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:57:26.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:57:26.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:26.687 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:57:26.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:57:26.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:26.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:26.963 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:26.963+0000 7fb0f6c18640 -1 Falling back to public interface 2024-06-03T06:57:26.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:27.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:27.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:27.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:57:27.976 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:57:27.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:27.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:28.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:29.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:29.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:29.275 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:57:29.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:57:29.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:29.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:29.401 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:29.401+0000 7fb0f6c18640 -1 osd.1 0 log_to_monitors true 2024-06-03T06:57:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:30.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:30.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:30.652 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:57:30.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:57:30.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:30.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:30.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:31.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:31.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:31.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:57:31.958 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:57:31.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:31.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:32.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:33.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:33.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:33.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:57:33.314 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:57:33.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:33.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:33.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:34.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:34.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:34.624 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:57:34.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:57:34.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:34.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:34.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:35.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:35.932 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:57:35.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:57:35.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:35.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:36.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:37.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:37.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:37.307 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:57:37.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:57:37.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:37.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:37.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:38.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:38.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:38.609 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:57:38.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:57:38.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:38.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:38.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:39.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:39.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:39.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:57:39.916 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:57:39.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:39.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:40.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:41.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:41.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:41.359 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:57:41.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:57:41.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:41.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:41.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:42.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:42.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:42.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:57:42.660 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:57:42.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:42.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:43.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:44.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:44.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:44.475 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:57:44.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:57:44.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:44.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:45.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:46.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:46.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:46.278 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:57:46.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:57:46.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:46.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:46.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:47.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:47.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:47.839 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:57:47.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:57:47.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:47.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:48.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:49.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:49.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:49.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:57:49.305 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:57:49.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:49.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:50.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:51.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:51.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:51.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:57:51.013 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:57:51.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:51.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:51.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:57:52.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:57:52.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:57:52.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:57:52.747 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:57:52.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:57:52.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:57:53.048 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4108610771,v1:127.0.0.1:6811/4108610771] [v2:127.0.0.1:6812/4108610771,v1:127.0.0.1:6813/4108610771] exists,up 82b55f3e-34c9-4014-a1f2-409ee6945bcd 2024-06-03T06:57:53.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:57:53.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:57:53.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:57:53.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-03T06:57:53.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-06-03T06:57:53.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T06:57:53.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T06:57:53.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T06:57:53.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T06:57:53.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:57:53.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:57:53.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:57:53.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:57:53.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:57:53.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:57:53.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:57:53.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T06:57:53.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T06:57:53.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:57:53.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T06:57:53.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:57:53.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:57:53.053 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:57:53.053 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:57:53.053 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:57:53.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:57:53.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:57:53.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:57:53.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T06:57:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T06:57:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:57:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:57:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:57:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:57:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:57:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:57:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T06:57:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T06:57:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:57:53.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T06:57:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=90548b71-2e41-4033-8329-9e0f3266684e 2024-06-03T06:57:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 90548b71-2e41-4033-8329-9e0f3266684e' 2024-06-03T06:57:53.055 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 90548b71-2e41-4033-8329-9e0f3266684e 2024-06-03T06:57:53.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T06:57:53.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxaV1mp2I2BBAAAe85Cs7pP78VjldgFI1PYQ== 2024-06-03T06:57:53.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxaV1mp2I2BBAAAe85Cs7pP78VjldgFI1PYQ=="}' 2024-06-03T06:57:53.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 90548b71-2e41-4033-8329-9e0f3266684e -i td/osd-scrub-repair/2/new.json 2024-06-03T06:57:53.441 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:57:53.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-03T06:57:53.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxaV1mp2I2BBAAAe85Cs7pP78VjldgFI1PYQ== --osd-uuid 90548b71-2e41-4033-8329-9e0f3266684e 2024-06-03T06:57:53.483 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:53.482+0000 7f9a1a207640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:57:53.534 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:53.533+0000 7f9a1a207640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:57:53.535 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:53.534+0000 7f9a1a207640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:57:53.536 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:53.535+0000 7f9a1a207640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:57:53.536 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:53.535+0000 7f9a1a207640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:57:53.536 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:53.535+0000 7f9a1a207640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T06:57:53.601 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:57:53.600+0000 7f9a1a207640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T06:57:59.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-03T06:57:59.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T06:57:59.952 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T06:57:59.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T06:57:59.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T06:58:00.326 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T06:58:00.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T06:58:00.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:58:00.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T06:58:00.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T06:58:00.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T06:58:00.353 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:58:00.353+0000 7fb08f568640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:58:00.355 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:58:00.356+0000 7fb08f568640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:58:00.356 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:58:00.357+0000 7fb08f568640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:58:00.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T06:58:00.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:58:00.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:58:00.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:58:00.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:58:00.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:00.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:58:00.616 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T06:58:00.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:00.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:00.871 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:58:00.871+0000 7fb08f568640 -1 Falling back to public interface 2024-06-03T06:58:00.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:01.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:01.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:01.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:58:01.904 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T06:58:01.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:01.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:02.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:03.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:03.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:03.205 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T06:58:03.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:58:03.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:03.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:03.339 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:58:03.339+0000 7fb08f568640 -1 osd.2 0 log_to_monitors true 2024-06-03T06:58:03.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:04.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:04.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:04.558 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T06:58:04.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:58:04.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:04.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:04.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:05.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:05.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:05.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T06:58:05.859 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T06:58:05.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:05.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:07.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:07.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:07.352 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T06:58:07.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T06:58:07.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:07.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:07.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:08.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:08.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:08.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T06:58:08.657 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T06:58:08.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:08.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:08.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:09.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:09.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:09.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T06:58:09.958 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T06:58:09.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:09.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:10.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:11.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:11.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:11.304 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T06:58:11.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T06:58:11.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:11.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:11.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:12.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:12.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:12.606 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T06:58:12.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T06:58:12.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:12.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:12.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:13.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:13.908 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T06:58:13.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:13.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T06:58:13.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:13.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:14.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:15.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:15.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:15.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T06:58:15.361 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T06:58:15.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:15.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:15.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:16.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:16.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:16.666 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T06:58:16.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T06:58:16.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:16.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:17.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T06:58:18.423 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T06:58:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:19.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:20.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:20.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:20.296 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T06:58:20.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T06:58:20.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:20.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:20.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:21.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:21.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:21.806 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T06:58:21.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T06:58:21.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:21.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:22.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T06:58:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T06:58:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:23.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:24.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:24.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:24.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T06:58:24.758 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T06:58:24.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:24.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:25.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:26.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:26.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:26.096 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T06:58:26.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T06:58:26.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:26.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:26.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:27.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:27.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:27.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T06:58:27.449 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T06:58:27.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:27.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:58:27.751 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/741385207,v1:127.0.0.1:6819/741385207] [v2:127.0.0.1:6820/741385207,v1:127.0.0.1:6821/741385207] exists,up 90548b71-2e41-4033-8329-9e0f3266684e 2024-06-03T06:58:27.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:58:27.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:58:27.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:58:27.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-06-03T06:58:27.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T06:58:28.046 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-06-03T06:58:28.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T06:58:28.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T06:58:28.593 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T06:58:28.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:58:29.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T06:58:31.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-06-03T06:58:31.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-06-03T06:58:32.053 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' created 2024-06-03T06:58:32.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:58:33.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-06-03T06:58:33.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-03T06:58:33.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-03T06:58:33.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-06-03T06:58:33.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-03T06:58:33.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-06-03T06:58:33.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-03T06:58:33.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-03T06:58:34.290 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'ecpool' created 2024-06-03T06:58:34.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T06:58:35.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-06-03T06:58:35.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-03T06:58:35.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:58:35.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:58:35.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:58:35.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:58:35.306 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:58:35.306 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:58:35.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:58:35.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:58:35.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:58:35.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:58:35.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:58:35.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:58:35.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:58:35.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:58:35.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:58:35.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:58:35.754 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:58:35.754 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:58:35.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:58:35.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:58:35.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:58:35.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-06-03T06:58:35.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-06-03T06:58:35.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-06-03T06:58:35.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:58:35.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:58:35.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-06-03T06:58:35.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-06-03T06:58:35.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672974' 2024-06-03T06:58:35.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:58:35.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:58:36.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-06-03T06:58:36.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-06-03T06:58:36.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672974 2-64424509447' 2024-06-03T06:58:36.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:58:36.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-06-03T06:58:36.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:58:36.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:58:36.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-06-03T06:58:36.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:58:36.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-06-03T06:58:36.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-06-03T06:58:36.122 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836502 2024-06-03T06:58:36.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:58:36.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-06-03T06:58:36.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:58:36.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-06-03T06:58:36.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:58:36.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:58:36.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-06-03T06:58:36.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:58:36.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-06-03T06:58:36.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-06-03T06:58:36.414 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672974 2024-06-03T06:58:36.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:58:36.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-06-03T06:58:36.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:58:36.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-06-03T06:58:36.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:58:36.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:58:36.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-06-03T06:58:36.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:58:36.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-06-03T06:58:36.708 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509447 2024-06-03T06:58:36.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-06-03T06:58:36.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:58:37.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509447 2024-06-03T06:58:37.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:58:38.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:58:38.013 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:58:38.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509447 2024-06-03T06:58:38.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:58:38.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:58:38.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:58:38.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T06:58:38.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:58:38.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:58:38.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:58:38.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:58:38.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:58:38.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:58:38.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:58:38.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T06:58:38.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:58:38.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:58:38.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:58:39.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T06:58:39.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:58:39.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:58:39.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-03T06:58:39.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-06-03T06:58:39.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:58:39.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:58:39.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:58:39.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:58:39.365 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:58:39.365 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:58:39.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:58:39.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:58:39.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:58:39.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:58:39.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:58:39.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:58:39.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:58:39.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:58:39.459 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:58:39.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:58:39.745 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:58:39.745 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:58:39.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:58:39.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:58:39.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:58:39.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-06-03T06:58:39.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-06-03T06:58:39.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-06-03T06:58:39.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:58:39.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:58:39.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-06-03T06:58:39.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-06-03T06:58:39.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672976' 2024-06-03T06:58:39.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:58:39.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:58:40.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-06-03T06:58:40.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-06-03T06:58:40.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672976 2-64424509449' 2024-06-03T06:58:40.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:58:40.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-06-03T06:58:40.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:58:40.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:58:40.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-06-03T06:58:40.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:58:40.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-06-03T06:58:40.073 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836504 2024-06-03T06:58:40.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-06-03T06:58:40.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:58:40.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-06-03T06:58:40.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:58:40.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-06-03T06:58:40.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:58:40.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:58:40.363 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-06-03T06:58:40.363 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:58:40.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-06-03T06:58:40.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-06-03T06:58:40.365 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672976 2024-06-03T06:58:40.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:58:40.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-06-03T06:58:40.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:58:40.660 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-06-03T06:58:40.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:58:40.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:58:40.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-06-03T06:58:40.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:58:40.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-06-03T06:58:40.663 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509449 2024-06-03T06:58:40.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-06-03T06:58:40.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:58:40.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509449 2024-06-03T06:58:40.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:58:41.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:58:41.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:58:42.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509449 2024-06-03T06:58:42.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:58:42.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:58:42.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:58:42.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T06:58:42.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:58:42.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:58:42.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:58:42.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:58:42.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:58:42.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:58:42.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:58:42.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T06:58:42.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:58:42.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:58:42.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:58:43.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T06:58:43.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:58:43.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:58:43.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-06-03T06:58:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-03T06:58:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-06-03T06:58:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-06-03T06:58:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T06:58:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T06:58:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-06-03T06:58:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T06:58:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T06:58:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T06:58:43.679 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T06:58:43.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T06:58:44.240 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T06:58:44.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T06:58:44.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T06:58:44.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:58:44.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-06-03T06:58:44.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-03T06:58:44.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-03T06:58:44.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-06-03T06:58:44.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-06-03T06:58:44.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T06:58:44.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:58:44.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:58:44.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:58:44.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:58:44.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T06:58:44.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:58:44.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:58:44.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:58:44.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:58:44.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:58:44.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:58:44.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:58:44.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:58:44.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:58:44.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:58:44.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:58:44.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T06:58:44.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:58:44.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:58:44.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:58:44.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:58:44.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:58:44.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T06:58:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:58:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:58:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:58:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:58:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:58:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:58:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:58:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:58:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:58:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:58:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:58:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:58:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:58:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:58:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:58:47.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:58:47.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:58:47.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:58:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:58:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:58:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:58:47.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:58:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:58:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:58:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:58:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:58:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:58:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:58:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:58:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:58:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:58:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:58:47.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:58:47.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:58:47.338 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:58:47.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:58:47.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:58:47.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:58:47.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:58:47.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:58:47.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:58:47.366 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:58:47.366+0000 7fded47b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:58:47.369 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:58:47.368+0000 7fded47b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:58:47.370 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:58:47.370+0000 7fded47b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:58:47.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:58:47.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:58:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:58:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:58:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:58:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:58:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:58:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:47.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:58:47.887 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:58:47.886+0000 7fded47b8640 -1 Falling back to public interface 2024-06-03T06:58:47.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:48.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:48.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:48.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:58:48.931 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:58:48.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:48.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:58:49.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:50.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:50.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:50.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:58:50.232 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:58:50.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:50.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:58:50.340 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:58:50.339+0000 7fded47b8640 -1 osd.1 35 log_to_monitors true 2024-06-03T06:58:50.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:58:51.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:58:51.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:58:51.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:58:51.552 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:58:51.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:58:51.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:58:51.855 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/955707527,v1:127.0.0.1:6811/955707527] [v2:127.0.0.1:6812/955707527,v1:127.0.0.1:6813/955707527] exists,up 82b55f3e-34c9-4014-a1f2-409ee6945bcd 2024-06-03T06:58:51.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:58:51.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:58:51.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:58:51.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:58:51.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:58:51.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:58:51.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:58:51.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:58:51.856 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:58:51.856 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:58:51.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:58:51.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:58:51.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:58:51.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:58:51.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:58:51.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:58:51.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:58:51.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:58:51.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:58:52.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:58:52.502 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:58:52.502 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:58:52.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:58:52.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:58:52.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:58:52.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-06-03T06:58:52.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-06-03T06:58:52.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-06-03T06:58:52.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:58:52.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:58:52.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-06-03T06:58:52.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-06-03T06:58:52.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-171798691842' 2024-06-03T06:58:52.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:58:52.783 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:58:52.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-06-03T06:58:52.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-06-03T06:58:52.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-171798691842 2-64424509453' 2024-06-03T06:58:52.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:58:52.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-06-03T06:58:52.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:58:52.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:58:52.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-06-03T06:58:52.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:58:52.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-06-03T06:58:52.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-06-03T06:58:52.894 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836507 2024-06-03T06:58:52.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:58:53.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-06-03T06:58:53.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:58:54.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:58:54.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:58:54.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836507 2024-06-03T06:58:54.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:58:54.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-06-03T06:58:54.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:58:54.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:58:54.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-06-03T06:58:54.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:58:54.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-06-03T06:58:54.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-06-03T06:58:54.490 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 171798691842 2024-06-03T06:58:54.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:58:54.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-06-03T06:58:54.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:58:54.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-06-03T06:58:54.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:58:54.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:58:54.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-06-03T06:58:54.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:58:54.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-06-03T06:58:54.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-06-03T06:58:54.781 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509453 2024-06-03T06:58:54.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:58:55.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-06-03T06:58:55.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:58:55.084 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:58:55.084 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:58:55.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T06:58:55.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:58:55.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:58:55.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:58:55.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:58:55.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:58:55.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:58:55.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:58:55.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T06:58:55.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:58:55.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:58:55.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:58:56.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T06:58:56.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:58:56.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:58:56.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-03T06:58:56.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-06-03T06:58:56.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-06-03T06:58:56.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T06:58:56.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T06:58:56.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-06-03T06:58:56.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T06:58:56.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T06:58:56.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T06:58:56.515 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T06:58:56.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T06:58:57.042 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T06:58:57.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T06:58:57.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T06:58:57.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:58:57.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-06-03T06:58:57.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-03T06:58:57.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-03T06:58:57.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-06-03T06:58:57.351 INFO:tasks.workunit.client.0.smithi039.stderr:1+0 records in 2024-06-03T06:58:57.352 INFO:tasks.workunit.client.0.smithi039.stderr:1+0 records out 2024-06-03T06:58:57.352 INFO:tasks.workunit.client.0.smithi039.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.5647e-05 s, 31.2 MB/s 2024-06-03T06:58:57.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T06:58:57.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:58:57.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:58:57.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:58:57.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:58:57.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T06:58:57.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:58:57.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:58:57.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:58:57.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:58:57.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:58:57.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:58:57.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:58:57.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:58:57.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:58:57.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:58:58.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:58:58.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T06:58:58.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:58:58.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:58:58.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:58:58.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:58:58.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:58:58.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T06:59:00.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T06:59:00.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:59:00.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:59:00.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:59:00.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:59:00.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:59:00.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:59:00.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:59:00.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:59:00.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:59:00.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:59:00.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:59:00.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:59:00.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:59:00.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:59:00.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:59:00.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:59:00.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:59:00.512 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:59:00.512 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:59:00.512 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:59:00.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:59:00.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:59:00.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:59:00.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:59:00.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:59:00.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:59:00.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:59:00.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:59:00.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:59:00.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:59:00.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:59:00.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:59:00.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:59:00.514 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:59:00.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:59:00.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:59:00.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:59:00.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:59:00.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:59:00.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:59:00.542 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:00.543+0000 7ff9de81d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:59:00.545 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:00.545+0000 7ff9de81d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:59:00.546 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:00.546+0000 7ff9de81d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:59:00.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:59:00.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:59:00.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:59:00.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:59:00.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:59:00.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:00.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:59:00.820 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:59:00.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:00.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:59:01.064 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:01.064+0000 7ff9de81d640 -1 Falling back to public interface 2024-06-03T06:59:01.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:59:02.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:59:02.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:02.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:59:02.156 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:59:02.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:02.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:59:02.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:59:03.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:59:03.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:03.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:59:03.489 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:59:03.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:03.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:59:03.576 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:03.576+0000 7ff9de81d640 -1 osd.0 44 log_to_monitors true 2024-06-03T06:59:03.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:59:04.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:59:04.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:04.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:59:04.793 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:59:04.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:04.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:59:05.097 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 48 up_thru 40 down_at 45 last_clean_interval [5,44) [v2:127.0.0.1:6802/3432361171,v1:127.0.0.1:6803/3432361171] [v2:127.0.0.1:6804/3432361171,v1:127.0.0.1:6805/3432361171] exists,up 6b7a2520-f2c7-451d-9603-1bd397e95ea5 2024-06-03T06:59:05.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:59:05.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:59:05.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:59:05.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:59:05.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:59:05.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:59:05.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:59:05.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:59:05.098 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:59:05.098 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:59:05.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:59:05.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:59:05.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:59:05.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:59:05.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:59:05.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:59:05.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:59:05.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:59:05.193 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:59:05.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:59:05.495 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:59:05.495 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:59:05.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:59:05.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:59:05.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:59:05.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430210 2024-06-03T06:59:05.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430210 2024-06-03T06:59:05.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210' 2024-06-03T06:59:05.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:59:05.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:59:05.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691846 2024-06-03T06:59:05.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691846 2024-06-03T06:59:05.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-171798691846' 2024-06-03T06:59:05.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:59:05.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:59:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-06-03T06:59:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-06-03T06:59:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-171798691846 2-64424509456' 2024-06-03T06:59:05.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:59:05.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430210 2024-06-03T06:59:05.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:59:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:59:05.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430210 2024-06-03T06:59:05.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:59:05.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430210 2024-06-03T06:59:05.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430210' 2024-06-03T06:59:05.959 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 206158430210 2024-06-03T06:59:05.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:59:06.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430210 -lt 206158430210 2024-06-03T06:59:06.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:59:06.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691846 2024-06-03T06:59:06.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:59:06.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:59:06.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691846 2024-06-03T06:59:06.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:59:06.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691846 2024-06-03T06:59:06.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691846' 2024-06-03T06:59:06.277 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 171798691846 2024-06-03T06:59:06.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:59:06.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691846 -lt 171798691846 2024-06-03T06:59:06.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:59:06.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-06-03T06:59:06.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:59:06.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:59:06.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-06-03T06:59:06.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:59:06.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-06-03T06:59:06.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-06-03T06:59:06.573 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509456 2024-06-03T06:59:06.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:59:06.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-06-03T06:59:06.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:59:06.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:59:06.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:59:07.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T06:59:07.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:59:07.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:59:07.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:59:07.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:59:07.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:59:07.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:59:07.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:59:07.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:59:07.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:59:07.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:59:07.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:59:07.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-03T06:59:07.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-06-03T06:59:07.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T06:59:07.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-06-03T06:59:07.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:59:07.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T06:59:07.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:59:07.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:59:07.984 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:59:07.984 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:59:07.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:59:07.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:59:07.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:59:07.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:59:08.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:59:08.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:59:08.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:59:08.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:59:08.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-03T06:59:08.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-06-03T06:59:08.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:59:08.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:59:08.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:59:08.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:59:08.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:59:08.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:59:08.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:59:09.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T06:59:09.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T06:59:09.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T06:59:09.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:59:09.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T06:59:09.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:59:09.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:59:09.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:59:09.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:59:09.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:59:09.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:59:09.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:59:09.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:59:09.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T06:59:09.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:59:09.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:59:09.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:59:09.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-03T06:59:09.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-06-03T06:59:09.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T06:59:09.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T06:59:09.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T06:59:09.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T06:59:09.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T06:59:09.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T06:59:09.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T06:59:10.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T06:59:10.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T06:59:10.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-03T06:59:10.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T06:59:10.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-03T06:59:10.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T06:59:10.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:59:10.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:59:10.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:59:10.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:59:10.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:59:10.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:59:10.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:59:10.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T06:59:10.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:59:10.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:59:10.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:59:11.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T06:59:11.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:59:11.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:59:11.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-03T06:59:11.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-06-03T06:59:11.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-06-03T06:59:11.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T06:59:11.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T06:59:11.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-06-03T06:59:11.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T06:59:11.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T06:59:11.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T06:59:11.722 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T06:59:11.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T06:59:12.149 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T06:59:12.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T06:59:12.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T06:59:12.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:59:12.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-06-03T06:59:12.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-03T06:59:12.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-03T06:59:12.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-06-03T06:59:12.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:59:12.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:59:12.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:59:12.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:59:12.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-06-03T06:59:12.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:59:12.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:59:12.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:59:12.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:59:12.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:59:12.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:59:12.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:59:12.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:59:12.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:59:12.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:59:13.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:59:13.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-06-03T06:59:13.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:59:13.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:59:13.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T06:59:13.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:59:13.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:59:13.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-06-03T06:59:15.066 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-06-03T06:59:15.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T06:59:15.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:59:15.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:59:15.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T06:59:15.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:59:15.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T06:59:15.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:59:15.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:59:15.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:59:15.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:59:15.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T06:59:15.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T06:59:15.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:59:15.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:59:15.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:59:15.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:59:15.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:59:15.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:59:15.586 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:59:15.586 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:59:15.586 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:59:15.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:59:15.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:59:15.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:59:15.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:59:15.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:59:15.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:59:15.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:59:15.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:59:15.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:59:15.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:59:15.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:59:15.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T06:59:15.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T06:59:15.588 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T06:59:15.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:59:15.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T06:59:15.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T06:59:15.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:59:15.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:59:15.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:59:15.615 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:15.615+0000 7f0cfa014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:59:15.618 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:15.618+0000 7f0cfa014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:59:15.619 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:15.619+0000 7f0cfa014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:59:15.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T06:59:15.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:59:15.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T06:59:15.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:59:15.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:59:15.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:15.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:59:15.890 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:59:15.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:15.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:59:16.134 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:16.135+0000 7f0cfa014640 -1 Falling back to public interface 2024-06-03T06:59:16.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:59:17.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:59:17.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:17.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:59:17.181 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:59:17.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:17.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:59:17.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:59:18.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:59:18.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:18.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:59:18.486 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:59:18.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:18.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:59:18.518 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:18.519+0000 7f0cfa014640 -1 osd.1 53 log_to_monitors true 2024-06-03T06:59:18.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:59:19.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:59:19.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:19.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:59:19.904 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:59:19.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:19.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T06:59:20.323 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 54 last_clean_interval [40,53) [v2:127.0.0.1:6810/2344392728,v1:127.0.0.1:6811/2344392728] [v2:127.0.0.1:6812/2344392728,v1:127.0.0.1:6813/2344392728] exists,up 82b55f3e-34c9-4014-a1f2-409ee6945bcd 2024-06-03T06:59:20.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:59:20.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:59:20.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:59:20.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:59:20.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:59:20.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:59:20.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:59:20.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:59:20.324 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:59:20.325 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:59:20.325 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:59:20.325 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:59:20.325 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:59:20.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:59:20.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:59:20.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:59:20.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:59:20.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:59:20.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:59:20.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:59:20.746 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:59:20.746 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:59:20.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:59:20.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:59:20.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:59:20.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430214 2024-06-03T06:59:20.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430214 2024-06-03T06:59:20.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430214' 2024-06-03T06:59:20.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:59:20.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:59:20.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103170 2024-06-03T06:59:20.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103170 2024-06-03T06:59:20.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430214 1-249108103170' 2024-06-03T06:59:20.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:59:20.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:59:21.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-06-03T06:59:21.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-06-03T06:59:21.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430214 1-249108103170 2-64424509460' 2024-06-03T06:59:21.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:59:21.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430214 2024-06-03T06:59:21.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:59:21.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:59:21.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430214 2024-06-03T06:59:21.073 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:59:21.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430214 2024-06-03T06:59:21.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430214' 2024-06-03T06:59:21.074 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 206158430214 2024-06-03T06:59:21.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:59:21.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430213 -lt 206158430214 2024-06-03T06:59:21.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:59:22.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:59:22.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:59:22.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430214 -lt 206158430214 2024-06-03T06:59:22.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:59:22.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103170 2024-06-03T06:59:22.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:59:22.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:59:22.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103170 2024-06-03T06:59:22.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:59:22.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103170 2024-06-03T06:59:22.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103170' 2024-06-03T06:59:22.670 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 249108103170 2024-06-03T06:59:22.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:59:23.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103170 -lt 249108103170 2024-06-03T06:59:23.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:59:23.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-06-03T06:59:23.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:59:23.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:59:23.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-06-03T06:59:23.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:59:23.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-06-03T06:59:23.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-06-03T06:59:23.087 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509460 2024-06-03T06:59:23.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:59:23.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509460 2024-06-03T06:59:23.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:59:23.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:59:23.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:59:23.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T06:59:23.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:59:23.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:59:23.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:59:23.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:59:23.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:59:23.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:59:23.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:59:24.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T06:59:24.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:59:24.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:59:24.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:59:24.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T06:59:24.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:59:24.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:59:24.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-03T06:59:24.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-06-03T06:59:24.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-06-03T06:59:24.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T06:59:24.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T06:59:24.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-06-03T06:59:24.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T06:59:24.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T06:59:24.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T06:59:24.848 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T06:59:24.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T06:59:25.392 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T06:59:25.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T06:59:25.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T06:59:25.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:59:25.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-06-03T06:59:25.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-03T06:59:25.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-03T06:59:25.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-06-03T06:59:25.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-06-03T06:59:25.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-06-03T06:59:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-03T06:59:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:59:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:59:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T06:59:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:59:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-03T06:59:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:59:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:59:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T06:59:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:59:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T06:59:25.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:59:25.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:59:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:59:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:59:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:59:26.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:59:26.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-03T06:59:26.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:59:26.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:59:26.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T06:59:26.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:59:26.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:59:26.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-03T06:59:28.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T06:59:28.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:59:28.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:59:28.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T06:59:28.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:59:28.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T06:59:28.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:59:28.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:59:28.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:59:28.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:59:28.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T06:59:28.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T06:59:28.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:59:28.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:59:28.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:59:28.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:59:28.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:59:28.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:59:28.554 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:59:28.554 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:59:28.554 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:59:28.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:59:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:59:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:59:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:59:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:59:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:59:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:59:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:59:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:59:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:59:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:59:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T06:59:28.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T06:59:28.556 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T06:59:28.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:59:28.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T06:59:28.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T06:59:28.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:59:28.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:59:28.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:59:28.585 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:28.585+0000 7f2fd2c0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:59:28.587 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:28.587+0000 7f2fd2c0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:59:28.588 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:28.589+0000 7f2fd2c0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:59:28.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T06:59:28.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:59:28.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T06:59:28.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:59:28.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:59:28.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:28.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:59:28.863 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:59:28.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:28.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:59:29.103 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:29.103+0000 7f2fd2c0c640 -1 Falling back to public interface 2024-06-03T06:59:29.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:59:30.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:59:30.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:30.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:59:30.177 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:59:30.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:30.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:59:30.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:59:31.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:59:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:59:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:59:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:59:31.543 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:31.543+0000 7f2fd2c0c640 -1 osd.0 62 log_to_monitors true 2024-06-03T06:59:31.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:59:32.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:59:32.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:32.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:59:32.930 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:59:32.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:32.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T06:59:33.235 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 67 up_thru 67 down_at 63 last_clean_interval [48,62) [v2:127.0.0.1:6802/3626298878,v1:127.0.0.1:6803/3626298878] [v2:127.0.0.1:6804/3626298878,v1:127.0.0.1:6805/3626298878] exists,up 6b7a2520-f2c7-451d-9603-1bd397e95ea5 2024-06-03T06:59:33.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:59:33.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:59:33.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:59:33.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:59:33.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:59:33.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:59:33.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:59:33.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:59:33.236 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:59:33.236 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:59:33.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:59:33.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:59:33.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:59:33.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:59:33.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:59:33.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:59:33.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:59:33.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:59:33.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:59:33.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:59:33.691 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:59:33.691 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:59:33.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:59:33.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:59:33.691 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:59:33.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-06-03T06:59:33.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-06-03T06:59:33.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834' 2024-06-03T06:59:33.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:59:33.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:59:33.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103174 2024-06-03T06:59:33.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103174 2024-06-03T06:59:33.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834 1-249108103174' 2024-06-03T06:59:33.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:59:33.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:59:34.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-06-03T06:59:34.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-06-03T06:59:34.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834 1-249108103174 2-64424509464' 2024-06-03T06:59:34.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:59:34.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808834 2024-06-03T06:59:34.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:59:34.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:59:34.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808834 2024-06-03T06:59:34.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:59:34.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-06-03T06:59:34.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808834' 2024-06-03T06:59:34.020 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 287762808834 2024-06-03T06:59:34.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:59:34.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808834 -lt 287762808834 2024-06-03T06:59:34.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:59:34.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103174 2024-06-03T06:59:34.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:59:34.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:59:34.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103174 2024-06-03T06:59:34.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:59:34.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103174 2024-06-03T06:59:34.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103174' 2024-06-03T06:59:34.313 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 249108103174 2024-06-03T06:59:34.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:59:34.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103174 -lt 249108103174 2024-06-03T06:59:34.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:59:34.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-06-03T06:59:34.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:59:34.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:59:34.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-06-03T06:59:34.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:59:34.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-06-03T06:59:34.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-06-03T06:59:34.603 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509464 2024-06-03T06:59:34.605 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:59:34.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-06-03T06:59:34.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:59:34.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:59:34.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:59:35.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T06:59:35.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:59:35.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:59:35.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:59:35.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:59:35.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:59:35.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:59:35.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:59:35.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T06:59:35.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:59:35.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:59:35.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:59:35.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T06:59:35.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:59:35.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:59:35.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-03T06:59:35.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:59:35.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:59:35.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-03T06:59:35.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:59:35.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-03T06:59:35.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:59:35.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:59:35.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-03T06:59:35.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:59:35.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-03T06:59:35.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:59:35.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:59:35.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:59:35.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:59:35.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:59:36.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:59:36.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-03T06:59:36.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:59:36.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:59:36.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-03T06:59:36.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:59:36.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:59:36.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-03T06:59:38.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-03T06:59:38.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:59:38.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:59:38.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-03T06:59:38.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:59:38.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:59:38.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:59:38.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:59:38.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:59:38.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:59:38.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:59:38.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:59:38.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:59:38.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:59:38.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:59:38.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:59:38.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:59:38.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:59:38.728 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:59:38.728 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:59:38.729 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:59:38.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:59:38.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:59:38.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:59:38.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:59:38.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:59:38.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:59:38.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:59:38.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:59:38.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:59:38.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:59:38.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:59:38.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:59:38.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-03T06:59:38.730 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.2 2024-06-03T06:59:38.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:59:38.730 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-03T06:59:38.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-03T06:59:38.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:59:38.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:59:38.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:59:38.757 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:38.757+0000 7f8761d66640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:59:38.760 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:38.759+0000 7f8761d66640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:59:38.762 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:38.761+0000 7f8761d66640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:59:39.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-03T06:59:39.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:59:39.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:59:39.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:59:39.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:59:39.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:39.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:59:39.055 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:59:39.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:39.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:59:39.280 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:39.280+0000 7f8761d66640 -1 Falling back to public interface 2024-06-03T06:59:39.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:59:40.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:59:40.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:40.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:59:40.343 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:59:40.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:40.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:59:40.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:59:41.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:59:41.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:41.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:59:41.649 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:59:41.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:41.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:59:41.712 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:41.711+0000 7f8761d66640 -1 osd.2 69 log_to_monitors true 2024-06-03T06:59:41.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:59:42.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:59:42.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:42.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:59:42.961 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:59:42.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:42.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:59:43.369 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/4069704160,v1:127.0.0.1:6819/4069704160] [v2:127.0.0.1:6820/4069704160,v1:127.0.0.1:6821/4069704160] exists,up 90548b71-2e41-4033-8329-9e0f3266684e 2024-06-03T06:59:43.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:59:43.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:59:43.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:59:43.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:59:43.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:59:43.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:59:43.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:59:43.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:59:43.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:59:43.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:59:43.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:59:43.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:59:43.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:59:43.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:59:43.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:59:43.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:59:43.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:59:43.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:59:43.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:59:43.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:59:43.753 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:59:43.753 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:59:43.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:59:43.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:59:43.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:59:43.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808837 2024-06-03T06:59:43.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808837 2024-06-03T06:59:43.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808837' 2024-06-03T06:59:43.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:59:43.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:59:43.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103177 2024-06-03T06:59:43.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103177 2024-06-03T06:59:43.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808837 1-249108103177' 2024-06-03T06:59:43.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:59:43.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:59:44.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-06-03T06:59:44.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-06-03T06:59:44.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808837 1-249108103177 2-317827579906' 2024-06-03T06:59:44.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:59:44.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808837 2024-06-03T06:59:44.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:59:44.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:59:44.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808837 2024-06-03T06:59:44.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:59:44.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808837 2024-06-03T06:59:44.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808837' 2024-06-03T06:59:44.080 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 287762808837 2024-06-03T06:59:44.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:59:44.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808837 -lt 287762808837 2024-06-03T06:59:44.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:59:44.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103177 2024-06-03T06:59:44.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:59:44.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:59:44.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103177 2024-06-03T06:59:44.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:59:44.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103177 2024-06-03T06:59:44.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103177' 2024-06-03T06:59:44.376 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 249108103177 2024-06-03T06:59:44.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:59:44.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103177 -lt 249108103177 2024-06-03T06:59:44.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:59:44.720 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579906 2024-06-03T06:59:44.720 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:59:44.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:59:44.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579906 2024-06-03T06:59:44.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:59:44.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-06-03T06:59:44.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-06-03T06:59:44.723 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 317827579906 2024-06-03T06:59:44.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:59:45.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 317827579906 2024-06-03T06:59:45.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:59:46.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:59:46.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:59:46.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579906 -lt 317827579906 2024-06-03T06:59:46.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:59:46.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:59:46.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:59:46.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T06:59:46.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:59:46.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:59:46.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:59:46.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:59:46.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:59:46.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:59:46.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:59:46.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T06:59:46.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:59:46.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:59:46.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:59:47.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T06:59:47.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:59:47.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:59:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-06-03T06:59:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-03T06:59:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:59:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:59:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-03T06:59:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:59:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-03T06:59:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:59:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:59:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-03T06:59:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:59:47.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-03T06:59:47.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:59:47.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:59:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:59:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:59:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T06:59:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T06:59:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-03T06:59:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T06:59:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T06:59:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-03T06:59:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T06:59:47.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:59:47.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-03T06:59:50.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-03T06:59:50.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T06:59:50.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T06:59:50.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-03T06:59:50.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T06:59:50.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T06:59:50.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T06:59:50.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T06:59:50.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T06:59:50.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T06:59:50.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T06:59:50.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T06:59:50.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T06:59:50.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T06:59:50.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T06:59:50.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T06:59:50.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T06:59:50.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T06:59:50.011 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T06:59:50.011 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T06:59:50.011 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T06:59:50.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:59:50.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T06:59:50.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T06:59:50.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T06:59:50.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T06:59:50.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T06:59:50.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T06:59:50.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T06:59:50.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T06:59:50.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T06:59:50.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T06:59:50.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T06:59:50.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-03T06:59:50.013 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.2 2024-06-03T06:59:50.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T06:59:50.013 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-03T06:59:50.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-03T06:59:50.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T06:59:50.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T06:59:50.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T06:59:50.042 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:50.042+0000 7fa4c9615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:59:50.044 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:50.045+0000 7fa4c9615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:59:50.046 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:50.046+0000 7fa4c9615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T06:59:50.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-03T06:59:50.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T06:59:50.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T06:59:50.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T06:59:50.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T06:59:50.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:50.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T06:59:50.314 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T06:59:50.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:50.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:59:50.558 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:50.559+0000 7fa4c9615640 -1 Falling back to public interface 2024-06-03T06:59:50.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:59:51.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:59:51.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:51.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T06:59:51.602 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:59:51.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:51.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:59:51.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:59:52.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:59:52.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:52.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T06:59:52.908 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T06:59:52.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:52.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:59:52.914 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T06:59:52.914+0000 7fa4c9615640 -1 osd.2 76 log_to_monitors true 2024-06-03T06:59:53.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T06:59:54.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T06:59:54.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T06:59:54.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T06:59:54.270 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T06:59:54.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T06:59:54.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T06:59:54.611 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 up in weight 1 up_from 81 up_thru 81 down_at 77 last_clean_interval [74,76) [v2:127.0.0.1:6818/2918662305,v1:127.0.0.1:6819/2918662305] [v2:127.0.0.1:6820/2918662305,v1:127.0.0.1:6821/2918662305] exists,up 90548b71-2e41-4033-8329-9e0f3266684e 2024-06-03T06:59:54.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T06:59:54.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T06:59:54.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T06:59:54.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T06:59:54.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T06:59:54.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T06:59:54.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T06:59:54.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T06:59:54.613 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T06:59:54.613 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T06:59:54.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T06:59:54.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T06:59:54.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T06:59:54.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T06:59:54.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T06:59:54.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T06:59:54.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T06:59:54.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T06:59:54.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T06:59:55.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T06:59:55.122 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T06:59:55.122 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T06:59:55.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T06:59:55.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:59:55.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T06:59:55.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808840 2024-06-03T06:59:55.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808840 2024-06-03T06:59:55.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808840' 2024-06-03T06:59:55.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:59:55.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T06:59:55.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103180 2024-06-03T06:59:55.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103180 2024-06-03T06:59:55.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808840 1-249108103180' 2024-06-03T06:59:55.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T06:59:55.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T06:59:55.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350978 2024-06-03T06:59:55.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350978 2024-06-03T06:59:55.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808840 1-249108103180 2-347892350978' 2024-06-03T06:59:55.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:59:55.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808840 2024-06-03T06:59:55.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:59:55.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T06:59:55.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808840 2024-06-03T06:59:55.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:59:55.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808840 2024-06-03T06:59:55.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808840' 2024-06-03T06:59:55.449 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 287762808840 2024-06-03T06:59:55.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:59:55.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808839 -lt 287762808840 2024-06-03T06:59:55.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T06:59:56.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T06:59:56.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T06:59:57.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808840 -lt 287762808840 2024-06-03T06:59:57.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:59:57.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103180 2024-06-03T06:59:57.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:59:57.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T06:59:57.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103180 2024-06-03T06:59:57.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:59:57.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103180 2024-06-03T06:59:57.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103180' 2024-06-03T06:59:57.051 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 249108103180 2024-06-03T06:59:57.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T06:59:57.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103180 -lt 249108103180 2024-06-03T06:59:57.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T06:59:57.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350978 2024-06-03T06:59:57.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T06:59:57.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T06:59:57.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350978 2024-06-03T06:59:57.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T06:59:57.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350978 2024-06-03T06:59:57.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350978' 2024-06-03T06:59:57.340 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 347892350978 2024-06-03T06:59:57.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T06:59:57.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350978 -lt 347892350978 2024-06-03T06:59:57.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T06:59:57.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:59:57.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:59:57.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T06:59:57.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T06:59:57.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T06:59:57.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T06:59:57.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T06:59:57.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T06:59:57.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T06:59:57.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T06:59:58.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T06:59:58.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T06:59:58.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T06:59:58.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T06:59:58.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T06:59:58.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T06:59:58.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T06:59:58.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-06-03T06:59:58.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-03T06:59:58.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-06-03T06:59:58.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-06-03T06:59:58.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T06:59:58.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T06:59:58.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-06-03T06:59:58.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T06:59:58.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T06:59:58.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T06:59:59.031 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T06:59:59.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T06:59:59.516 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T06:59:59.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T06:59:59.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T06:59:59.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-06-03T06:59:59.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-06-03T06:59:59.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-03T06:59:59.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-03T06:59:59.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-06-03T06:59:59.757 INFO:tasks.workunit.client.0.smithi039.stderr:2+0 records in 2024-06-03T06:59:59.758 INFO:tasks.workunit.client.0.smithi039.stderr:2+0 records out 2024-06-03T06:59:59.758 INFO:tasks.workunit.client.0.smithi039.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 7.088e-05 s, 57.8 MB/s 2024-06-03T06:59:59.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T06:59:59.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T06:59:59.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T06:59:59.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T06:59:59.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T06:59:59.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T06:59:59.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T06:59:59.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T06:59:59.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T06:59:59.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T06:59:59.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T06:59:59.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T06:59:59.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T06:59:59.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T06:59:59.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T06:59:59.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:00:01.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:00:01.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:00:01.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:00:01.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:00:01.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:00:01.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:00:01.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:00:01.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:00:02.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:00:02.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:00:02.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:00:02.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:00:02.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:00:02.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:00:02.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:00:02.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:00:02.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:00:02.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:00:02.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:00:02.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:00:02.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:00:02.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:00:02.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:00:02.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:00:02.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:00:02.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:00:02.768 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:00:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:00:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:00:02.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:00:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:00:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:00:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:00:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:00:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:00:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:00:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:00:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:00:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:00:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:00:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:00:02.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:00:02.771 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:00:02.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:00:02.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:00:02.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:00:02.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:00:02.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:00:02.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:00:02.799 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:02.799+0000 7f2826415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:00:02.801 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:02.801+0000 7f2826415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:00:02.802 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:02.803+0000 7f2826415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:00:03.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:00:03.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:00:03.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:00:03.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:00:03.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:00:03.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:03.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:00:03.113 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:00:03.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:03.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:00:03.314 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:03.315+0000 7f2826415640 -1 Falling back to public interface 2024-06-03T07:00:03.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:00:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:00:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:00:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:00:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:00:04.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:00:05.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:00:05.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:05.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:00:05.711 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:00:05.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:05.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:00:05.736 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:05.737+0000 7f2826415640 -1 osd.1 85 log_to_monitors true 2024-06-03T07:00:06.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:00:07.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:00:07.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:07.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:00:07.046 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:00:07.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:07.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:00:07.417 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 90 up_thru 90 down_at 86 last_clean_interval [58,85) [v2:127.0.0.1:6810/331486638,v1:127.0.0.1:6811/331486638] [v2:127.0.0.1:6812/331486638,v1:127.0.0.1:6813/331486638] exists,up 82b55f3e-34c9-4014-a1f2-409ee6945bcd 2024-06-03T07:00:07.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:00:07.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:00:07.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:00:07.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:00:07.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:00:07.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:00:07.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:00:07.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:00:07.418 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:00:07.418 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:00:07.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:00:07.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:00:07.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:00:07.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:00:07.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:00:07.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:00:07.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:00:07.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:00:07.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:00:07.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:00:07.803 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:00:07.803 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:00:07.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:00:07.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:00:07.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:00:07.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808844 2024-06-03T07:00:07.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808844 2024-06-03T07:00:07.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808844' 2024-06-03T07:00:07.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:00:07.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:00:08.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056642 2024-06-03T07:00:08.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056642 2024-06-03T07:00:08.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808844 1-386547056642' 2024-06-03T07:00:08.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:00:08.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:00:08.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350982 2024-06-03T07:00:08.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350982 2024-06-03T07:00:08.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808844 1-386547056642 2-347892350982' 2024-06-03T07:00:08.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:00:08.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808844 2024-06-03T07:00:08.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:00:08.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:00:08.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808844 2024-06-03T07:00:08.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:00:08.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808844 2024-06-03T07:00:08.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808844' 2024-06-03T07:00:08.133 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 287762808844 2024-06-03T07:00:08.133 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:00:08.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808844 -lt 287762808844 2024-06-03T07:00:08.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:00:08.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056642 2024-06-03T07:00:08.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:00:08.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:00:08.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056642 2024-06-03T07:00:08.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:00:08.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056642 2024-06-03T07:00:08.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2024-06-03T07:00:08.425 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 386547056642 2024-06-03T07:00:08.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:00:08.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056642 -lt 386547056642 2024-06-03T07:00:08.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:00:08.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350982 2024-06-03T07:00:08.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:00:08.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:00:08.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350982 2024-06-03T07:00:08.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:00:08.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350982 2024-06-03T07:00:08.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350982' 2024-06-03T07:00:08.717 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 347892350982 2024-06-03T07:00:08.717 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:00:09.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350980 -lt 347892350982 2024-06-03T07:00:09.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:00:10.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:00:10.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:00:10.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350982 -lt 347892350982 2024-06-03T07:00:10.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:00:10.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:00:10.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:00:10.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:00:10.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:00:10.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:00:10.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:00:10.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:00:10.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:00:10.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:00:10.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:00:10.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:00:10.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:00:10.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:00:10.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:00:11.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:00:11.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:00:11.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:00:11.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-03T07:00:11.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-06-03T07:00:11.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-06-03T07:00:11.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:00:11.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T07:00:11.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-06-03T07:00:11.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:00:11.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:00:11.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:00:11.713 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:00:11.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:00:12.266 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:00:12.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:00:12.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:00:12.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:00:12.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-06-03T07:00:12.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-03T07:00:12.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-03T07:00:12.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-03T07:00:12.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:00:12.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:00:12.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:00:12.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:00:12.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-03T07:00:12.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:00:12.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:00:12.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:00:12.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:00:12.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:00:12.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:00:12.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:00:12.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:00:12.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:00:12.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:00:12.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:00:12.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-03T07:00:12.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:00:12.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:00:12.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:00:12.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:00:12.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:00:12.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-06-03T07:00:15.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:00:15.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:00:15.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:00:15.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:00:15.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:00:15.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:00:15.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:00:15.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:00:15.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:00:15.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:00:15.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:00:15.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:00:15.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:00:15.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:00:15.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:00:15.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:00:15.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:00:15.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:00:15.023 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:00:15.023 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:00:15.023 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:00:15.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:00:15.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:00:15.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:00:15.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:00:15.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:00:15.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:00:15.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:00:15.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:00:15.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:00:15.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:00:15.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:00:15.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:00:15.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:00:15.025 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:00:15.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:00:15.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:00:15.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:00:15.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:00:15.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:00:15.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:00:15.054 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:15.054+0000 7f5fbd015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:00:15.056 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:15.056+0000 7f5fbd015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:00:15.057 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:15.057+0000 7f5fbd015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:00:15.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:00:15.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:00:15.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:00:15.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:00:15.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:00:15.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:15.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:00:15.325 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:00:15.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:15.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:00:15.572 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:15.572+0000 7f5fbd015640 -1 Falling back to public interface 2024-06-03T07:00:15.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:00:16.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:00:16.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:16.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:00:16.617 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:00:16.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:16.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:00:16.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:00:17.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:00:17.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:17.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:00:17.923 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:00:17.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:17.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:00:17.984 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:17.984+0000 7f5fbd015640 -1 osd.0 94 log_to_monitors true 2024-06-03T07:00:18.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:00:19.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:00:19.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:19.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:00:19.303 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:00:19.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:19.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:00:19.605 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 99 up_thru 90 down_at 95 last_clean_interval [67,94) [v2:127.0.0.1:6802/869323495,v1:127.0.0.1:6803/869323495] [v2:127.0.0.1:6804/869323495,v1:127.0.0.1:6805/869323495] exists,up 6b7a2520-f2c7-451d-9603-1bd397e95ea5 2024-06-03T07:00:19.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:00:19.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:00:19.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:00:19.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:00:19.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:00:19.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:00:19.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:00:19.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:00:19.607 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:00:19.607 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:00:19.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:00:19.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:00:19.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:00:19.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:00:19.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:00:19.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:00:19.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:00:19.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:00:19.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:00:19.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:00:19.991 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:00:19.991 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:00:19.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:00:19.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:00:19.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:00:20.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-06-03T07:00:20.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-06-03T07:00:20.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762306' 2024-06-03T07:00:20.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:00:20.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:00:20.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056645 2024-06-03T07:00:20.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056645 2024-06-03T07:00:20.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762306 1-386547056645' 2024-06-03T07:00:20.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:00:20.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:00:20.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350985 2024-06-03T07:00:20.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350985 2024-06-03T07:00:20.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762306 1-386547056645 2-347892350985' 2024-06-03T07:00:20.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:00:20.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-425201762306 2024-06-03T07:00:20.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:00:20.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:00:20.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-425201762306 2024-06-03T07:00:20.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:00:20.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-06-03T07:00:20.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 425201762306' 2024-06-03T07:00:20.419 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 425201762306 2024-06-03T07:00:20.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:00:20.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 425201762306 2024-06-03T07:00:20.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:00:21.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:00:21.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:00:22.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 425201762306 2024-06-03T07:00:22.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:00:23.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T07:00:23.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:00:23.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762306 -lt 425201762306 2024-06-03T07:00:23.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:00:23.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056645 2024-06-03T07:00:23.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:00:23.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:00:23.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056645 2024-06-03T07:00:23.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:00:23.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056645 2024-06-03T07:00:23.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056645' 2024-06-03T07:00:23.323 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 386547056645 2024-06-03T07:00:23.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:00:23.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056646 -lt 386547056645 2024-06-03T07:00:23.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:00:23.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350985 2024-06-03T07:00:23.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:00:23.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:00:23.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350985 2024-06-03T07:00:23.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:00:23.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350985 2024-06-03T07:00:23.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350985' 2024-06-03T07:00:23.621 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 347892350985 2024-06-03T07:00:23.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:00:23.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350985 -lt 347892350985 2024-06-03T07:00:23.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:00:23.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:00:23.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:00:24.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:00:24.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:00:24.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:00:24.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:00:24.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:00:24.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:00:24.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:00:24.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:00:24.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:00:24.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:00:24.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:00:24.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:00:24.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:00:24.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:00:24.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:00:24.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-06-03T07:00:24.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-03T07:00:24.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:00:24.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:00:24.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:00:24.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:00:24.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-03T07:00:24.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:00:24.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:00:24.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:00:24.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:00:24.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:00:24.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:00:24.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:00:24.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:00:24.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:00:24.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:00:25.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:00:25.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-03T07:00:25.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:00:25.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:00:25.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:00:25.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:00:25.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:00:25.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-03T07:00:27.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:00:27.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:00:27.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:00:27.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:00:27.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:00:27.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:00:27.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:00:27.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:00:27.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:00:27.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:00:27.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:00:27.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:00:27.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:00:27.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:00:27.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:00:27.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:00:27.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:00:27.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:00:27.375 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:00:27.375 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:00:27.376 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:00:27.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:00:27.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:00:27.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:00:27.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:00:27.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:00:27.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:00:27.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:00:27.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:00:27.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:00:27.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:00:27.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:00:27.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:00:27.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:00:27.378 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:00:27.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:00:27.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:00:27.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:00:27.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:00:27.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:00:27.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:00:27.406 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:27.405+0000 7f1ca6a26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:00:27.409 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:27.408+0000 7f1ca6a26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:00:27.410 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:27.409+0000 7f1ca6a26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:00:27.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:00:27.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:00:27.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:00:27.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:00:27.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:00:27.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:27.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:00:27.679 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:00:27.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:27.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:00:27.933 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:27.933+0000 7f1ca6a26640 -1 Falling back to public interface 2024-06-03T07:00:27.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:00:28.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:00:28.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:28.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:00:28.966 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:00:28.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:28.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:00:29.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:00:30.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:00:30.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:30.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:00:30.321 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:00:30.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:30.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:00:30.448 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:30.447+0000 7f1ca6a26640 -1 osd.1 101 log_to_monitors true 2024-06-03T07:00:30.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:00:31.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:00:31.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:31.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:00:31.671 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:00:31.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:31.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:00:32.044 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 106 up_thru 106 down_at 102 last_clean_interval [90,101) [v2:127.0.0.1:6810/3661158444,v1:127.0.0.1:6811/3661158444] [v2:127.0.0.1:6812/3661158444,v1:127.0.0.1:6813/3661158444] exists,up 82b55f3e-34c9-4014-a1f2-409ee6945bcd 2024-06-03T07:00:32.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:00:32.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:00:32.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:00:32.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:00:32.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:00:32.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:00:32.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:00:32.045 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:00:32.046 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:00:32.046 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:00:32.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:00:32.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:00:32.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:00:32.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:00:32.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:00:32.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:00:32.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:00:32.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:00:32.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:00:32.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:00:32.432 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:00:32.432 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:00:32.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:00:32.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:00:32.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:00:32.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762309 2024-06-03T07:00:32.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762309 2024-06-03T07:00:32.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762309' 2024-06-03T07:00:32.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:00:32.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:00:32.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533378 2024-06-03T07:00:32.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533378 2024-06-03T07:00:32.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762309 1-455266533378' 2024-06-03T07:00:32.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:00:32.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:00:32.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350988 2024-06-03T07:00:32.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350988 2024-06-03T07:00:32.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762309 1-455266533378 2-347892350988' 2024-06-03T07:00:32.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:00:32.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-425201762309 2024-06-03T07:00:32.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:00:32.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:00:32.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-425201762309 2024-06-03T07:00:32.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:00:32.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762309 2024-06-03T07:00:32.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 425201762309' 2024-06-03T07:00:32.853 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 425201762309 2024-06-03T07:00:32.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:00:33.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762308 -lt 425201762309 2024-06-03T07:00:33.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:00:34.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:00:34.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:00:34.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762310 -lt 425201762309 2024-06-03T07:00:34.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:00:34.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533378 2024-06-03T07:00:34.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:00:34.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:00:34.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533378 2024-06-03T07:00:34.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:00:34.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533378 2024-06-03T07:00:34.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533378' 2024-06-03T07:00:34.448 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 455266533378 2024-06-03T07:00:34.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:00:34.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533378 -lt 455266533378 2024-06-03T07:00:34.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:00:34.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350988 2024-06-03T07:00:34.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:00:34.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:00:34.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350988 2024-06-03T07:00:34.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:00:34.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350988 2024-06-03T07:00:34.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350988' 2024-06-03T07:00:34.743 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 347892350988 2024-06-03T07:00:34.743 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:00:35.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350989 -lt 347892350988 2024-06-03T07:00:35.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:00:35.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:00:35.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:00:35.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:00:35.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:00:35.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:00:35.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:00:35.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:00:35.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:00:35.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:00:35.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:00:35.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:00:35.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:00:35.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:00:35.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:00:36.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:00:36.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:00:36.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:00:36.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-03T07:00:36.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-06-03T07:00:36.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-06-03T07:00:36.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:00:36.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T07:00:36.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-06-03T07:00:36.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:00:36.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:00:36.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:00:36.503 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:00:36.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:00:36.904 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:00:36.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:00:36.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:00:36.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:00:37.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-06-03T07:00:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-03T07:00:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-03T07:00:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-06-03T07:00:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-06-03T07:00:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-06-03T07:00:37.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-03T07:00:37.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:00:37.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:00:37.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:00:37.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:00:37.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-03T07:00:37.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:00:37.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:00:37.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:00:37.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:00:37.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:00:37.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:00:37.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:00:37.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:00:37.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:00:37.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:00:37.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:00:37.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-03T07:00:37.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:00:37.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:00:37.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:00:37.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:00:37.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:00:37.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-06-03T07:00:39.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:00:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:00:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:00:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:00:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:00:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:00:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:00:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:00:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:00:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:00:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:00:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:00:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:00:39.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:00:39.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:00:39.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:00:39.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:00:39.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:00:39.672 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:00:39.672 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:00:39.672 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:00:39.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:00:39.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:00:39.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:00:39.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:00:39.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:00:39.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:00:39.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:00:39.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:00:39.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:00:39.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:00:39.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:00:39.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:00:39.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:00:39.674 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:00:39.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:00:39.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:00:39.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:00:39.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:00:39.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:00:39.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:00:39.704 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:39.704+0000 7fd41ae32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:00:39.706 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:39.706+0000 7fd41ae32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:00:39.707 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:39.708+0000 7fd41ae32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:00:39.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:00:39.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:00:39.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:00:39.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:00:39.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:00:39.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:39.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:00:39.973 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:00:39.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:39.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:00:40.225 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:40.226+0000 7fd41ae32640 -1 Falling back to public interface 2024-06-03T07:00:40.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:00:41.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:00:41.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:41.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:00:41.266 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:00:41.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:41.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:00:41.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:00:42.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:00:42.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:42.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:00:42.567 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:00:42.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:42.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:00:42.710 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:42.710+0000 7fd41ae32640 -1 osd.0 111 log_to_monitors true 2024-06-03T07:00:42.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:00:43.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:00:43.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:43.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:00:43.870 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:00:43.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:43.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:00:44.370 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 116 up_thru 116 down_at 112 last_clean_interval [99,111) [v2:127.0.0.1:6802/4005138202,v1:127.0.0.1:6803/4005138202] [v2:127.0.0.1:6804/4005138202,v1:127.0.0.1:6805/4005138202] exists,up 6b7a2520-f2c7-451d-9603-1bd397e95ea5 2024-06-03T07:00:44.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:00:44.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:00:44.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:00:44.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:00:44.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:00:44.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:00:44.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:00:44.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:00:44.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:00:44.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:00:44.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:00:44.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:00:44.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:00:44.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:00:44.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:00:44.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:00:44.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:00:44.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:00:44.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:00:44.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:00:44.796 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:00:44.796 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:00:44.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:00:44.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:00:44.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:00:44.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-06-03T07:00:44.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-06-03T07:00:44.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338' 2024-06-03T07:00:44.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:00:44.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:00:45.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533381 2024-06-03T07:00:45.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533381 2024-06-03T07:00:45.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338 1-455266533381' 2024-06-03T07:00:45.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:00:45.035 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:00:45.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350992 2024-06-03T07:00:45.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350992 2024-06-03T07:00:45.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338 1-455266533381 2-347892350992' 2024-06-03T07:00:45.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:00:45.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-498216206338 2024-06-03T07:00:45.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:00:45.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:00:45.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-498216206338 2024-06-03T07:00:45.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:00:45.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-06-03T07:00:45.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 498216206338' 2024-06-03T07:00:45.145 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 498216206338 2024-06-03T07:00:45.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:00:45.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 498216206338 2024-06-03T07:00:45.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:00:46.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:00:46.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:00:46.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206338 -lt 498216206338 2024-06-03T07:00:46.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:00:46.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533381 2024-06-03T07:00:46.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:00:46.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:00:46.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533381 2024-06-03T07:00:46.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:00:46.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533381 2024-06-03T07:00:46.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533381' 2024-06-03T07:00:46.735 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 455266533381 2024-06-03T07:00:46.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:00:47.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533382 -lt 455266533381 2024-06-03T07:00:47.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:00:47.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350992 2024-06-03T07:00:47.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:00:47.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:00:47.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350992 2024-06-03T07:00:47.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:00:47.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350992 2024-06-03T07:00:47.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350992' 2024-06-03T07:00:47.028 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 347892350992 2024-06-03T07:00:47.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:00:47.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350992 -lt 347892350992 2024-06-03T07:00:47.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:00:47.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:00:47.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:00:47.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:00:47.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:00:47.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:00:47.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:00:47.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:00:47.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:00:47.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:00:47.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:00:47.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:00:47.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:00:47.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:00:47.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:00:48.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:00:48.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:00:48.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:00:48.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-03T07:00:48.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:00:48.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:00:48.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:00:48.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:00:48.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-03T07:00:48.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:00:48.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:00:48.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:00:48.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:00:48.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:00:48.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:00:48.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:00:48.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:00:48.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:00:48.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:00:48.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:00:48.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-03T07:00:48.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:00:48.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:00:48.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:00:48.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:00:48.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:00:48.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-03T07:00:50.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:00:50.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:00:50.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:00:50.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:00:50.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:00:50.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:00:50.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:00:50.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:00:50.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:00:50.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:00:50.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:00:50.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:00:50.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:00:50.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:00:50.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:00:50.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:00:50.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:00:50.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:00:50.917 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:00:50.917 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:00:50.917 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:00:50.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:00:50.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:00:50.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:00:50.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:00:50.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:00:50.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:00:50.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:00:50.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:00:50.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:00:50.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:00:50.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:00:50.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:00:50.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:00:50.918 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:00:50.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:00:50.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:00:50.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:00:50.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:00:50.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:00:50.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:00:50.944 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:50.944+0000 7f5c62599640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:00:50.946 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:50.946+0000 7f5c62599640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:00:50.947 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:50.948+0000 7f5c62599640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:00:51.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:00:51.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:00:51.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:00:51.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:00:51.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:00:51.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:51.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:00:51.219 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:00:51.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:51.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:00:51.460 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:51.460+0000 7f5c62599640 -1 Falling back to public interface 2024-06-03T07:00:51.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:00:52.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:00:52.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:52.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:00:52.508 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:00:52.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:52.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:00:52.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:00:53.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:00:53.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:53.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:00:53.813 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:00:53.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:53.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:00:53.814 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:00:53.814+0000 7f5c62599640 -1 osd.0 118 log_to_monitors true 2024-06-03T07:00:54.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:00:55.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:00:55.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:00:55.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:00:55.172 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:00:55.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:00:55.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:00:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 123 up_thru 123 down_at 119 last_clean_interval [116,118) [v2:127.0.0.1:6802/3066932259,v1:127.0.0.1:6803/3066932259] [v2:127.0.0.1:6804/3066932259,v1:127.0.0.1:6805/3066932259] exists,up 6b7a2520-f2c7-451d-9603-1bd397e95ea5 2024-06-03T07:00:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:00:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:00:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:00:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:00:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:00:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:00:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:00:55.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:00:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:00:55.559 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:00:55.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:00:55.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:00:55.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:00:55.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:00:55.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:00:55.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:00:55.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:00:55.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:00:55.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:00:55.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:00:55.945 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:00:55.945 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:00:55.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:00:55.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:00:55.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:00:56.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977410 2024-06-03T07:00:56.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977410 2024-06-03T07:00:56.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-528280977410' 2024-06-03T07:00:56.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:00:56.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:00:56.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533385 2024-06-03T07:00:56.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533385 2024-06-03T07:00:56.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-528280977410 1-455266533385' 2024-06-03T07:00:56.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:00:56.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:00:56.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350995 2024-06-03T07:00:56.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350995 2024-06-03T07:00:56.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-528280977410 1-455266533385 2-347892350995' 2024-06-03T07:00:56.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:00:56.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-528280977410 2024-06-03T07:00:56.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:00:56.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:00:56.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-528280977410 2024-06-03T07:00:56.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:00:56.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977410 2024-06-03T07:00:56.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 528280977410' 2024-06-03T07:00:56.336 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 528280977410 2024-06-03T07:00:56.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:00:56.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977410 -lt 528280977410 2024-06-03T07:00:56.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:00:56.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533385 2024-06-03T07:00:56.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:00:56.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:00:56.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533385 2024-06-03T07:00:56.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:00:56.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533385 2024-06-03T07:00:56.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533385' 2024-06-03T07:00:56.627 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 455266533385 2024-06-03T07:00:56.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:00:56.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533384 -lt 455266533385 2024-06-03T07:00:56.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:00:57.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:00:57.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:00:58.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533385 -lt 455266533385 2024-06-03T07:00:58.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:00:58.217 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350995 2024-06-03T07:00:58.217 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:00:58.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:00:58.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350995 2024-06-03T07:00:58.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:00:58.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350995 2024-06-03T07:00:58.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350995' 2024-06-03T07:00:58.220 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 347892350995 2024-06-03T07:00:58.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:00:58.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350995 -lt 347892350995 2024-06-03T07:00:58.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:00:58.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:00:58.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:00:58.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:00:58.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:00:58.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:00:58.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:00:58.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:00:58.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:00:58.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:00:58.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:00:59.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:00:59.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:00:59.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:00:59.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:00:59.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:00:59.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:00:59.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:00:59.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-06-03T07:00:59.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-06-03T07:00:59.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T07:00:59.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-06-03T07:00:59.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-06-03T07:00:59.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T07:00:59.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-06-03T07:00:59.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-06-03T07:00:59.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-06-03T07:00:59.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-06-03T07:00:59.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:00:59.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:00:59.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:00:59.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:01:00.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-06-03T06:58:34.206882+0000 2024-06-03T07:01:00.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-06-03T07:01:00.442 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-06-03T07:01:00.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-06-03T06:58:34.206882+0000 2024-06-03T07:01:00.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-03T07:01:00.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:58:34.206882+0000 2024-06-03T07:01:00.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T07:01:00.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:01:00.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:01:00.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:01:00.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:01:00.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:01:00.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:01:00.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:01:00.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:58:34.206882+0000 '>' 2024-06-03T06:58:34.206882+0000 2024-06-03T07:01:00.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:01:01.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:01:01.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:01:01.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:01:01.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:01:01.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:01:01.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:01:01.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:01:02.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:58:34.206882+0000 '>' 2024-06-03T06:58:34.206882+0000 2024-06-03T07:01:02.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:01:03.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:01:03.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:01:03.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:01:03.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:01:03.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:01:03.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:01:03.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:01:03.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:58:34.206882+0000 '>' 2024-06-03T06:58:34.206882+0000 2024-06-03T07:01:03.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:01:04.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:01:04.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:01:04.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:01:04.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:01:04.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:01:04.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:01:04.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:01:04.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:58:34.206882+0000 '>' 2024-06-03T06:58:34.206882+0000 2024-06-03T07:01:04.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:01:05.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:01:05.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:01:05.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:01:05.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:01:05.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:01:05.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:01:05.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:01:05.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:58:34.206882+0000 '>' 2024-06-03T06:58:34.206882+0000 2024-06-03T07:01:05.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:01:06.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:01:06.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:01:06.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:01:06.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:01:06.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:01:06.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:01:06.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:01:07.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:01:01.172372+0000 '>' 2024-06-03T06:58:34.206882+0000 2024-06-03T07:01:07.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:01:07.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-06-03T07:01:07.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-06-03T07:01:07.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-06-03T07:01:07.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-03T07:01:07.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-06-03T07:01:07.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-06-03T07:01:07.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-06-03T07:01:07.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=123 2024-06-03T07:01:07.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-03T07:01:07.317 INFO:tasks.workunit.client.0.smithi039.stderr: . as $in 2024-06-03T07:01:07.317 INFO:tasks.workunit.client.0.smithi039.stderr: | if type == "object" then 2024-06-03T07:01:07.317 INFO:tasks.workunit.client.0.smithi039.stderr: reduce keys[] as $key 2024-06-03T07:01:07.317 INFO:tasks.workunit.client.0.smithi039.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-03T07:01:07.317 INFO:tasks.workunit.client.0.smithi039.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-03T07:01:07.318 INFO:tasks.workunit.client.0.smithi039.stderr: else f 2024-06-03T07:01:07.318 INFO:tasks.workunit.client.0.smithi039.stderr: end; 2024-06-03T07:01:07.318 INFO:tasks.workunit.client.0.smithi039.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-03T07:01:07.318 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-03T07:01:07.318 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-03T07:01:07.318 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-03T07:01:07.318 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-03T07:01:07.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-06-03T07:01:07.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-03T07:01:07.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-03T07:01:07.341 INFO:tasks.workunit.client.0.smithi039.stderr: . as $in 2024-06-03T07:01:07.341 INFO:tasks.workunit.client.0.smithi039.stderr: | if type == "object" then 2024-06-03T07:01:07.341 INFO:tasks.workunit.client.0.smithi039.stderr: reduce keys[] as $key 2024-06-03T07:01:07.341 INFO:tasks.workunit.client.0.smithi039.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-03T07:01:07.341 INFO:tasks.workunit.client.0.smithi039.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-03T07:01:07.341 INFO:tasks.workunit.client.0.smithi039.stderr: else f 2024-06-03T07:01:07.341 INFO:tasks.workunit.client.0.smithi039.stderr: end; 2024-06-03T07:01:07.341 INFO:tasks.workunit.client.0.smithi039.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-03T07:01:07.342 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-03T07:01:07.342 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-03T07:01:07.342 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-03T07:01:07.342 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-03T07:01:07.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-06-03T07:01:07.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-03T07:01:07.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-03T07:01:07.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-03T07:01:07.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-06-03T07:01:07.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-06-03T07:01:07.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-06-03T07:01:07.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-06-03T07:01:07.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:01:07.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:01:07.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:01:07.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:01:07.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:01:07.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-06-03T06:58:34.206882+0000 2024-06-03T07:01:07.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-06-03T07:01:07.923 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-06-03T07:01:07.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-06-03T06:58:34.206882+0000 last_deep_scrub_stamp 2024-06-03T07:01:07.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-03T07:01:07.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T06:58:34.206882+0000 2024-06-03T07:01:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-06-03T07:01:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:01:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:01:07.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:01:07.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:01:07.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:01:07.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:01:07.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:01:08.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:58:34.206882+0000 '>' 2024-06-03T06:58:34.206882+0000 2024-06-03T07:01:08.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:01:09.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:01:09.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:01:09.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:01:09.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:01:09.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:01:09.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:01:09.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:01:09.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:58:34.206882+0000 '>' 2024-06-03T06:58:34.206882+0000 2024-06-03T07:01:09.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:01:10.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:01:10.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:01:10.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:01:10.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:01:10.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:01:10.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:01:10.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:01:10.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T06:58:34.206882+0000 '>' 2024-06-03T06:58:34.206882+0000 2024-06-03T07:01:10.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:01:11.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:01:11.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:01:11.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:01:11.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:01:11.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:01:11.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:01:11.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:01:12.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:01:08.347800+0000 '>' 2024-06-03T06:58:34.206882+0000 2024-06-03T07:01:12.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:01:12.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-06-03T07:01:12.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-06-03T07:01:12.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-06-03T07:01:12.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-03T07:01:12.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-06-03T07:01:12.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-06-03T07:01:12.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-06-03T07:01:12.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=123 2024-06-03T07:01:12.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-06-03T07:01:12.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-03T07:01:12.205 INFO:tasks.workunit.client.0.smithi039.stderr: . as $in 2024-06-03T07:01:12.205 INFO:tasks.workunit.client.0.smithi039.stderr: | if type == "object" then 2024-06-03T07:01:12.205 INFO:tasks.workunit.client.0.smithi039.stderr: reduce keys[] as $key 2024-06-03T07:01:12.205 INFO:tasks.workunit.client.0.smithi039.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-03T07:01:12.205 INFO:tasks.workunit.client.0.smithi039.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-03T07:01:12.205 INFO:tasks.workunit.client.0.smithi039.stderr: else f 2024-06-03T07:01:12.206 INFO:tasks.workunit.client.0.smithi039.stderr: end; 2024-06-03T07:01:12.206 INFO:tasks.workunit.client.0.smithi039.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-03T07:01:12.206 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-03T07:01:12.206 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-03T07:01:12.206 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-03T07:01:12.206 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-03T07:01:12.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-06-03T07:01:12.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-03T07:01:12.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-03T07:01:12.230 INFO:tasks.workunit.client.0.smithi039.stderr: . as $in 2024-06-03T07:01:12.230 INFO:tasks.workunit.client.0.smithi039.stderr: | if type == "object" then 2024-06-03T07:01:12.230 INFO:tasks.workunit.client.0.smithi039.stderr: reduce keys[] as $key 2024-06-03T07:01:12.230 INFO:tasks.workunit.client.0.smithi039.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-03T07:01:12.230 INFO:tasks.workunit.client.0.smithi039.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-03T07:01:12.230 INFO:tasks.workunit.client.0.smithi039.stderr: else f 2024-06-03T07:01:12.230 INFO:tasks.workunit.client.0.smithi039.stderr: end; 2024-06-03T07:01:12.230 INFO:tasks.workunit.client.0.smithi039.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-03T07:01:12.230 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-03T07:01:12.231 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-03T07:01:12.231 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-03T07:01:12.231 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-03T07:01:12.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-06-03T07:01:12.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-03T07:01:12.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-03T07:01:12.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-03T07:01:12.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-06-03T07:01:12.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-06-03T07:01:12.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-06-03T07:01:12.641 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'ecpool' removed 2024-06-03T07:01:12.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T07:01:12.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:01:12.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:01:12.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:01:12.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:01:12.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:01:12.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:01:12.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:01:12.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:01:12.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:01:12.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:01:12.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:01:12.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:01:12.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:01:12.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:01:12.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:01:12.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:01:12.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:01:12.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:01:12.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:01:12.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:01:12.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:01:12.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:01:12.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:01:12.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:01:12.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:01:12.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:01:12.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:01:12.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:01:12.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T07:01:12.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T07:01:12.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T07:01:12.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T07:01:12.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:01:12.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:01:12.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:01:12.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:01:12.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:01:12.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:01:12.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:01:12.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:01:12.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:01:12.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:01:12.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:01:12.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:01:12.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:01:12.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:01:12.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:01:12.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:01:12.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:01:12.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:01:12.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:01:12.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:01:12.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:01:12.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:01:12.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:01:12.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:01:12.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:01:12.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:01:12.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:01:12.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:01:12.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T07:01:12.810 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T07:01:12.810 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:01:12.810 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:01:12.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T07:01:12.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T07:01:12.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T07:01:12.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T07:01:12.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T07:01:12.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-06-03T07:01:12.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-06-03T07:01:12.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-06-03T07:01:12.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-06-03T07:01:12.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-06-03T07:01:12.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-06-03T07:01:12.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-06-03T07:01:12.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-06-03T07:01:12.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T07:01:12.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T07:01:12.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T07:01:12.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T07:01:12.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T07:01:12.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T07:01:13.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T07:01:13.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:01:13.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:01:13.111 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:01:13.111 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:01:13.111 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:01:13.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:01:13.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T07:01:13.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T07:01:13.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T07:01:13.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:01:13.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:01:13.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T07:01:13.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T07:01:13.304 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:01:13.304 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:01:13.304 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:01:13.304 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:01:13.304 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:01:13.304 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:01:13.304 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:01:13.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:01:13.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T07:01:13.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T07:01:13.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:01:13.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:01:13.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T07:01:13.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T07:01:13.389 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:01:13.389 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:01:13.389 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:01:13.390 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:01:13.390 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:01:13.390 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:01:13.390 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:01:13.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:01:13.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T07:01:13.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-06-03T07:01:13.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T07:01:13.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T07:01:13.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T07:01:13.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T07:01:13.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T07:01:13.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T07:01:14.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T07:01:14.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:01:14.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:01:14.287 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:01:14.287 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:01:14.287 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:01:14.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:01:14.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:01:14.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:01:14.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-06-03T07:01:14.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-03T07:01:14.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-06-03T07:01:14.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:01:14.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:01:14.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T07:01:14.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:01:14.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:01:14.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:01:14.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:01:14.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:01:14.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:01:14.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:01:14.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:01:14.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:01:14.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:01:14.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:01:14.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:01:14.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:01:14.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:01:14.322 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:01:14.322 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:01:14.322 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:01:14.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:01:14.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:01:14.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:01:14.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:01:14.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:01:14.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:01:14.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:01:14.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:01:14.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:01:14.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:01:14.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:01:14.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:01:14.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T07:01:14.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:01:14.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:01:14.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8cc55336-1d3d-4454-b6f9-edab91bdf7f3 2024-06-03T07:01:14.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8cc55336-1d3d-4454-b6f9-edab91bdf7f3' 2024-06-03T07:01:14.324 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 8cc55336-1d3d-4454-b6f9-edab91bdf7f3 2024-06-03T07:01:14.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:01:14.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA6al1m2/tMFBAAetjag5xvMO5GCNlhL0ksSA== 2024-06-03T07:01:14.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA6al1m2/tMFBAAetjag5xvMO5GCNlhL0ksSA=="}' 2024-06-03T07:01:14.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8cc55336-1d3d-4454-b6f9-edab91bdf7f3 -i td/osd-scrub-repair/0/new.json 2024-06-03T07:01:14.576 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:01:14.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T07:01:14.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6al1m2/tMFBAAetjag5xvMO5GCNlhL0ksSA== --osd-uuid 8cc55336-1d3d-4454-b6f9-edab91bdf7f3 2024-06-03T07:01:14.617 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:14.617+0000 7fb97ae18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:01:14.627 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:14.627+0000 7fb97ae18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:01:14.628 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:14.629+0000 7fb97ae18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:01:14.629 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:14.629+0000 7fb97ae18640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:01:14.629 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:14.629+0000 7fb97ae18640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:01:14.629 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:14.629+0000 7fb97ae18640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:01:14.719 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:14.719+0000 7fb97ae18640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T07:01:20.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T07:01:20.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:01:20.202 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T07:01:20.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T07:01:20.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:01:20.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T07:01:20.579 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T07:01:20.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:01:20.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:01:20.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:01:20.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:01:20.606 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:20.605+0000 7f219b6a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:01:20.609 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:20.609+0000 7f219b6a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:01:20.611 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:20.610+0000 7f219b6a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:01:20.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T07:01:20.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:01:20.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:01:20.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:01:20.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:01:20.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:20.887 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:01:20.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:01:20.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:20.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:21.144 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:21.143+0000 7f219b6a2640 -1 Falling back to public interface 2024-06-03T07:01:21.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:22.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:22.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:22.175 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:01:22.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:01:22.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:22.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:22.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:23.419 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:23.418+0000 7f219b6a2640 -1 osd.0 0 log_to_monitors true 2024-06-03T07:01:23.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:23.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:23.476 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:01:23.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:01:23.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:23.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:23.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:24.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:24.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:24.830 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:01:24.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:01:24.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:24.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:26.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:26.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:01:26.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:01:26.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:26.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:26.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:27.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:27.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:27.442 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:01:27.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:01:27.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:27.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:27.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:28.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:28.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:28.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:01:28.745 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:01:28.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:28.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:29.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:30.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:30.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:30.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:01:30.096 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:01:30.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:30.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:30.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:31.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:31.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:31.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:01:31.399 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:01:31.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:31.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:31.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:32.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:32.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:32.702 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:01:32.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:01:32.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:32.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:33.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:34.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:34.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:34.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:01:34.010 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:01:34.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:34.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:34.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:35.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:35.564 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:01:35.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:35.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:01:35.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:35.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:36.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:37.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:37.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:37.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:01:37.078 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:01:37.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:37.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:37.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:38.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:38.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:38.633 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:01:38.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:01:38.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:38.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:39.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:40.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:40.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:40.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:01:40.258 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:01:40.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:40.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:40.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:41.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:41.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:41.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:01:41.627 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:01:41.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:41.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:42.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:43.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:43.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:43.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:01:43.272 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:01:43.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:43.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:43.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:44.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:44.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:44.900 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:01:44.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:01:44.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:44.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:01:45.197 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3099036176,v1:127.0.0.1:6803/3099036176] [v2:127.0.0.1:6804/3099036176,v1:127.0.0.1:6805/3099036176] exists,up 8cc55336-1d3d-4454-b6f9-edab91bdf7f3 2024-06-03T07:01:45.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:01:45.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:01:45.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:01:45.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-03T07:01:45.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-06-03T07:01:45.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:01:45.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:01:45.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T07:01:45.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:01:45.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:01:45.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:01:45.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:01:45.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:01:45.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:01:45.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:01:45.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:01:45.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:01:45.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:01:45.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:01:45.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:01:45.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:01:45.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:01:45.202 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:01:45.202 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:01:45.202 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:01:45.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:01:45.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:01:45.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:01:45.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:01:45.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:01:45.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:01:45.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:01:45.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:01:45.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:01:45.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:01:45.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:01:45.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:01:45.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T07:01:45.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:01:45.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:01:45.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e0c162d7-c974-42b9-b532-ed08abf9157d 2024-06-03T07:01:45.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e0c162d7-c974-42b9-b532-ed08abf9157d' 2024-06-03T07:01:45.204 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 e0c162d7-c974-42b9-b532-ed08abf9157d 2024-06-03T07:01:45.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:01:45.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZal1mZy4fDRAAFlw765VwYA26MkpfdmVKKg== 2024-06-03T07:01:45.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZal1mZy4fDRAAFlw765VwYA26MkpfdmVKKg=="}' 2024-06-03T07:01:45.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e0c162d7-c974-42b9-b532-ed08abf9157d -i td/osd-scrub-repair/1/new.json 2024-06-03T07:01:45.769 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:01:45.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T07:01:45.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZal1mZy4fDRAAFlw765VwYA26MkpfdmVKKg== --osd-uuid e0c162d7-c974-42b9-b532-ed08abf9157d 2024-06-03T07:01:45.812 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:45.812+0000 7f0428612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:01:45.846 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:45.846+0000 7f0428612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:01:45.847 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:45.847+0000 7f0428612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:01:45.847 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:45.847+0000 7f0428612640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:01:45.847 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:45.847+0000 7f0428612640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:01:45.847 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:45.847+0000 7f0428612640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:01:45.908 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:45.908+0000 7f0428612640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T07:01:51.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T07:01:51.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:01:51.622 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T07:01:51.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T07:01:51.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:01:52.008 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T07:01:52.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T07:01:52.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:01:52.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:01:52.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:01:52.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:01:52.036 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:52.036+0000 7fa53b78e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:01:52.038 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:52.038+0000 7fa53b78e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:01:52.039 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:52.040+0000 7fa53b78e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:01:52.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T07:01:52.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:01:52.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:01:52.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:01:52.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:01:52.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:52.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:01:52.298 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:01:52.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:52.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:01:52.565 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:52.566+0000 7fa53b78e640 -1 Falling back to public interface 2024-06-03T07:01:52.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:53.589 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:01:53.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:01:53.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:53.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:01:53.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:54.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:54.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:54.893 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:01:54.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:01:54.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:54.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:01:54.964 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:01:54.964+0000 7fa53b78e640 -1 osd.1 0 log_to_monitors true 2024-06-03T07:01:55.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:56.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:56.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:56.249 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:01:56.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:01:56.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:56.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:01:56.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:57.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:57.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:57.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:01:57.555 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:01:57.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:57.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:01:57.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:01:58.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:01:58.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:01:58.930 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:01:58.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:01:58.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:01:58.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:01:59.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:00.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:00.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:00.230 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:02:00.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:02:00.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:00.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:02:00.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:01.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:01.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:01.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:02:01.543 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:02:01.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:01.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:02:01.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:02.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:02.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:02.948 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:02:02.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:02:02.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:02.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:02:03.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:04.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:04.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:04.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:02:04.249 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:02:04.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:04.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:02:04.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:05.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:05.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:05.552 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:02:05.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:02:05.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:05.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:02:05.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:06.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:06.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:06.852 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:02:06.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:02:06.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:06.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:02:07.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:08.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:08.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:08.547 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:02:08.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:02:08.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:08.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:02:09.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:10.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:10.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:10.149 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:02:10.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:02:10.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:10.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:02:10.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:11.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:11.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:11.686 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:02:11.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:02:11.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:11.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:02:12.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:13.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:13.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:13.296 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:02:13.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:02:13.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:13.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:02:13.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:14.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:14.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:14.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:02:14.645 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:02:14.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:14.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:02:15.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:16.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:16.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:16.314 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:02:16.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:02:16.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:16.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:02:16.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:17.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:17.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:17.658 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T07:02:17.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T07:02:17.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:17.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:02:17.955 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3672197902,v1:127.0.0.1:6811/3672197902] [v2:127.0.0.1:6812/3672197902,v1:127.0.0.1:6813/3672197902] exists,up e0c162d7-c974-42b9-b532-ed08abf9157d 2024-06-03T07:02:17.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:02:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:02:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:02:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-03T07:02:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-06-03T07:02:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:02:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:02:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T07:02:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:02:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T07:02:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:02:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:02:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:02:17.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:02:17.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T07:02:17.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T07:02:17.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:02:17.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:02:17.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:02:17.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:02:17.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:02:17.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:02:17.961 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:02:17.961 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:02:17.961 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:02:17.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:02:17.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:02:17.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:02:17.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:02:17.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:02:17.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:02:17.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:02:17.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:02:17.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:02:17.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:02:17.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:02:17.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:02:17.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T07:02:17.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T07:02:17.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:02:17.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2dd04ed2-911c-48e6-9324-0875996ba9ca 2024-06-03T07:02:17.963 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 2dd04ed2-911c-48e6-9324-0875996ba9ca 2024-06-03T07:02:17.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2dd04ed2-911c-48e6-9324-0875996ba9ca' 2024-06-03T07:02:17.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:02:17.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5al1mj5VhOhAAeflDK7de79hRtHL9Sj6efg== 2024-06-03T07:02:17.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5al1mj5VhOhAAeflDK7de79hRtHL9Sj6efg=="}' 2024-06-03T07:02:17.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2dd04ed2-911c-48e6-9324-0875996ba9ca -i td/osd-scrub-repair/2/new.json 2024-06-03T07:02:18.394 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:02:18.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-03T07:02:18.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5al1mj5VhOhAAeflDK7de79hRtHL9Sj6efg== --osd-uuid 2dd04ed2-911c-48e6-9324-0875996ba9ca 2024-06-03T07:02:18.435 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:02:18.436+0000 7f5c2ce2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:02:18.479 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:02:18.479+0000 7f5c2ce2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:02:18.480 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:02:18.480+0000 7f5c2ce2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:02:18.480 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:02:18.480+0000 7f5c2ce2e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T07:02:18.480 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:02:18.480+0000 7f5c2ce2e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T07:02:18.480 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:02:18.480+0000 7f5c2ce2e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T07:02:18.555 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:02:18.555+0000 7f5c2ce2e640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T07:02:24.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-03T07:02:24.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:02:24.551 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T07:02:24.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T07:02:24.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:02:24.925 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T07:02:24.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T07:02:24.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:02:24.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:02:24.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:02:24.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:02:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:02:24.953+0000 7f5f4fa98640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:02:24.955 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:02:24.955+0000 7f5f4fa98640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:02:24.956 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:02:24.956+0000 7f5f4fa98640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:02:25.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T07:02:25.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:02:25.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T07:02:25.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:02:25.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:02:25.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:25.216 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:02:25.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:02:25.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:25.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:25.469 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:02:25.469+0000 7f5f4fa98640 -1 Falling back to public interface 2024-06-03T07:02:25.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:26.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:26.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:26.540 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:02:26.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:02:26.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:26.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:26.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:27.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:27.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:27.843 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:02:27.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:02:27.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:27.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:02:27.926+0000 7f5f4fa98640 -1 osd.2 0 log_to_monitors true 2024-06-03T07:02:28.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:29.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:29.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:02:29.146 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:02:29.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:29.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:29.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:30.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:30.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:30.466 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:02:30.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:02:30.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:30.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:30.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:31.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:31.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:31.788 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:02:31.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:02:31.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:31.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:32.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:33.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:33.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:33.089 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:02:33.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:02:33.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:33.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:33.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:34.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:34.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:34.391 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:02:34.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:02:34.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:34.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:34.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:35.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:35.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:35.806 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:02:35.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:02:35.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:35.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:36.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:37.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:37.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:37.187 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:02:37.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:02:37.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:37.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:37.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:38.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:38.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:38.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:02:38.493 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:02:38.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:38.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:38.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:39.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:39.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:39.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:02:39.821 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:02:39.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:39.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:40.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:41.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:41.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:41.205 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:02:41.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:02:41.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:41.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:41.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:42.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:42.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:42.958 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:02:42.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:02:42.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:42.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:43.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:44.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:44.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:44.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:02:44.924 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:02:44.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:44.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:45.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:46.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:46.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:46.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:02:46.334 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:02:46.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:46.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:46.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:47.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:47.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:47.745 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:02:47.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:02:47.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:47.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:48.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:49.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:49.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:49.323 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:02:49.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:02:49.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:49.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:49.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:02:50.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:02:50.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:02:50.950 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T07:02:50.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T07:02:50.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:02:50.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:02:51.249 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4080841341,v1:127.0.0.1:6819/4080841341] [v2:127.0.0.1:6820/4080841341,v1:127.0.0.1:6821/4080841341] exists,up 2dd04ed2-911c-48e6-9324-0875996ba9ca 2024-06-03T07:02:51.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:02:51.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:02:51.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:02:51.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-06-03T07:02:51.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T07:02:51.611 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-06-03T07:02:51.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T07:02:51.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T07:02:52.019 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T07:02:52.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:02:53.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T07:02:55.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-06-03T07:02:55.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-06-03T07:02:55.774 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' created 2024-06-03T07:02:55.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:02:56.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-06-03T07:02:56.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-03T07:02:56.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-03T07:02:56.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-06-03T07:02:56.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-03T07:02:56.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-06-03T07:02:57.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-03T07:02:57.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-03T07:02:58.134 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'ecpool' created 2024-06-03T07:02:58.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:02:59.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-06-03T07:02:59.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-06-03T07:02:59.715 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 3 allow_ec_overwrites to true 2024-06-03T07:02:59.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-03T07:02:59.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:02:59.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:02:59.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:02:59.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:02:59.728 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:02:59.728 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:02:59.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:02:59.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:02:59.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:02:59.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:02:59.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:02:59.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:02:59.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:02:59.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:02:59.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:03:00.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:03:00.197 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:03:00.197 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:03:00.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:03:00.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:00.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:03:00.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-06-03T07:03:00.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-06-03T07:03:00.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-06-03T07:03:00.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:00.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:03:00.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-06-03T07:03:00.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-06-03T07:03:00.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672972' 2024-06-03T07:03:00.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:00.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:03:00.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-06-03T07:03:00.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-06-03T07:03:00.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672972 2-64424509447' 2024-06-03T07:03:00.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:00.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-06-03T07:03:00.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:00.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:03:00.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-06-03T07:03:00.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:00.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-06-03T07:03:00.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-06-03T07:03:00.552 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836502 2024-06-03T07:03:00.552 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:03:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-06-03T07:03:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:00.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-06-03T07:03:00.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:00.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:03:00.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-06-03T07:03:00.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:00.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-06-03T07:03:00.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-06-03T07:03:00.849 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672972 2024-06-03T07:03:00.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:03:01.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-06-03T07:03:01.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:01.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-06-03T07:03:01.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:01.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:03:01.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-06-03T07:03:01.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:01.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-06-03T07:03:01.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-06-03T07:03:01.139 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509447 2024-06-03T07:03:01.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:03:01.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509447 2024-06-03T07:03:01.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:03:02.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:03:02.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:03:02.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509447 2024-06-03T07:03:02.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:03:02.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:03:02.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:03:03.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:03:03.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:03:03.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:03:03.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:03:03.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:03:03.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:03:03.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:03:03.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:03:03.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:03:03.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:03:03.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:03:03.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:03:03.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:03:03.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:03:03.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:03:03.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-03T07:03:03.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-06-03T07:03:03.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:03:03.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:03:03.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:03:03.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:03:03.812 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:03:03.812 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:03:03.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:03:03.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:03:03.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:03:03.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:03:03.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:03:03.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:03:03.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:03:03.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:03:03.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:03:04.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:03:04.195 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:03:04.195 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:03:04.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:03:04.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:04.195 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:03:04.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-06-03T07:03:04.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-06-03T07:03:04.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-06-03T07:03:04.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:04.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:03:04.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-06-03T07:03:04.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-06-03T07:03:04.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672974' 2024-06-03T07:03:04.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:04.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:03:04.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-06-03T07:03:04.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-06-03T07:03:04.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672974 2-64424509449' 2024-06-03T07:03:04.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:04.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-06-03T07:03:04.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:04.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:03:04.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-06-03T07:03:04.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:04.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-06-03T07:03:04.524 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836503 2024-06-03T07:03:04.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-06-03T07:03:04.525 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:03:04.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-06-03T07:03:04.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:04.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-06-03T07:03:04.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:04.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:03:04.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-06-03T07:03:04.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:04.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-06-03T07:03:04.826 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672974 2024-06-03T07:03:04.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-06-03T07:03:04.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:03:05.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-06-03T07:03:05.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:05.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-06-03T07:03:05.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:05.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:03:05.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-06-03T07:03:05.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:05.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-06-03T07:03:05.134 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509449 2024-06-03T07:03:05.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-06-03T07:03:05.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:03:05.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509449 2024-06-03T07:03:05.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:03:06.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:03:06.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:03:06.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509449 2024-06-03T07:03:06.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:03:06.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:03:06.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:03:07.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:03:07.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:03:07.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:03:07.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:03:07.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:03:07.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:03:07.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:03:07.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:03:07.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:03:07.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:03:07.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:03:07.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:03:07.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:03:07.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:03:07.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:03:07.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-06-03T07:03:07.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-03T07:03:07.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-06-03T07:03:07.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-06-03T07:03:07.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:03:07.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T07:03:07.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-06-03T07:03:07.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:03:07.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:03:07.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:03:08.186 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:03:08.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:03:08.637 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:03:08.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:03:08.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:03:08.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:03:08.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-06-03T07:03:08.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-03T07:03:08.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-03T07:03:08.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-06-03T07:03:08.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-06-03T07:03:08.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:03:08.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:03:08.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:03:08.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:03:08.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:03:08.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:03:08.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:03:08.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:03:08.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:03:08.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:03:08.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:03:08.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:03:08.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:03:08.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:03:08.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:03:08.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:03:09.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:03:09.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:03:09.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:03:09.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:03:09.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:03:09.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:03:09.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:03:09.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:03:11.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:03:11.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:03:11.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:03:11.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:03:11.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:03:11.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:03:11.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:03:11.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:03:11.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:03:11.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:03:11.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:03:11.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:03:11.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:03:11.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:03:11.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:03:11.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:03:11.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:03:11.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:03:11.492 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:03:11.492 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:03:11.492 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:03:11.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:03:11.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:03:11.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:03:11.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:03:11.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:03:11.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:03:11.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:03:11.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:03:11.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:03:11.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:03:11.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:03:11.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:03:11.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:03:11.494 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:03:11.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:03:11.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:03:11.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:03:11.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:03:11.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:03:11.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:03:11.524 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:11.524+0000 7f2fde3b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:03:11.526 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:11.527+0000 7f2fde3b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:03:11.528 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:11.528+0000 7f2fde3b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:03:11.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:03:11.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:03:11.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:03:11.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:03:11.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:03:11.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:03:11.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:03:11.796 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:03:11.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:03:11.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:03:12.043 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:12.043+0000 7f2fde3b8640 -1 Falling back to public interface 2024-06-03T07:03:12.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:03:13.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:03:13.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:03:13.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:03:13.084 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:03:13.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:03:13.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:03:13.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:03:14.381 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:14.381+0000 7f2fde3b8640 -1 osd.1 37 log_to_monitors true 2024-06-03T07:03:14.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:03:14.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:03:14.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:03:14.384 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:03:14.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:03:14.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:03:14.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:03:15.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:03:15.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:03:15.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:03:15.758 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:03:15.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:03:15.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:03:16.062 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 38 last_clean_interval [10,37) [v2:127.0.0.1:6810/390900946,v1:127.0.0.1:6811/390900946] [v2:127.0.0.1:6812/390900946,v1:127.0.0.1:6813/390900946] exists,up e0c162d7-c974-42b9-b532-ed08abf9157d 2024-06-03T07:03:16.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:03:16.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:03:16.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:03:16.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:03:16.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:03:16.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:03:16.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:03:16.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:03:16.064 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:03:16.064 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:03:16.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:03:16.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:03:16.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:03:16.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:03:16.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:03:16.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:03:16.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:03:16.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:03:16.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:03:16.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:03:16.453 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:03:16.453 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:03:16.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:03:16.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:16.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:03:16.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-06-03T07:03:16.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-06-03T07:03:16.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-06-03T07:03:16.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:16.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:03:16.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626434 2024-06-03T07:03:16.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626434 2024-06-03T07:03:16.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-180388626434' 2024-06-03T07:03:16.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:16.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:03:16.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-06-03T07:03:16.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-06-03T07:03:16.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-180388626434 2-64424509453' 2024-06-03T07:03:16.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:16.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-06-03T07:03:16.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:16.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:03:16.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-06-03T07:03:16.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:16.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-06-03T07:03:16.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-06-03T07:03:16.782 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836507 2024-06-03T07:03:16.783 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:03:17.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-06-03T07:03:17.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:03:18.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:03:18.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:03:18.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-06-03T07:03:18.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:03:19.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T07:03:19.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:03:19.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-06-03T07:03:19.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:19.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-180388626434 2024-06-03T07:03:19.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:19.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:03:19.698 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-180388626434 2024-06-03T07:03:19.698 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:19.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626434 2024-06-03T07:03:19.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 180388626434' 2024-06-03T07:03:19.699 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 180388626434 2024-06-03T07:03:19.699 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:03:20.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626434 -lt 180388626434 2024-06-03T07:03:20.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:20.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-06-03T07:03:20.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:20.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:03:20.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-06-03T07:03:20.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:20.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-06-03T07:03:20.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-06-03T07:03:20.033 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509453 2024-06-03T07:03:20.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:03:20.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-06-03T07:03:20.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:03:20.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:03:20.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:03:20.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:03:20.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:03:20.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:03:20.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:03:20.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:03:20.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:03:20.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:03:20.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:03:21.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:03:21.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:03:21.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:03:21.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:03:21.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:03:21.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:03:21.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:03:21.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-03T07:03:21.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-06-03T07:03:21.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-06-03T07:03:21.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:03:21.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T07:03:21.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-06-03T07:03:21.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:03:21.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:03:21.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:03:21.787 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:03:21.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:03:22.314 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:03:22.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:03:22.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:03:22.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:03:22.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-06-03T07:03:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-03T07:03:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-03T07:03:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-06-03T07:03:22.557 INFO:tasks.workunit.client.0.smithi039.stderr:1+0 records in 2024-06-03T07:03:22.557 INFO:tasks.workunit.client.0.smithi039.stderr:1+0 records out 2024-06-03T07:03:22.557 INFO:tasks.workunit.client.0.smithi039.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.8928e-05 s, 29.7 MB/s 2024-06-03T07:03:22.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:03:22.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:03:22.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:03:22.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:03:22.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:03:22.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:03:22.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:03:22.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:03:22.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:03:22.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:03:22.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:03:22.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:03:22.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:03:22.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:03:22.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:03:22.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:03:22.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:03:22.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:03:22.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:03:22.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:03:22.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:03:22.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:03:22.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:03:22.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:03:25.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:03:25.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:03:25.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:03:25.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:03:25.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:03:25.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:03:25.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:03:25.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:03:25.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:03:25.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:03:25.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:03:25.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:03:25.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:03:25.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:03:25.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:03:25.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:03:25.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:03:25.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:03:25.197 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:03:25.197 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:03:25.197 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:03:25.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:03:25.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:03:25.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:03:25.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:03:25.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:03:25.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:03:25.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:03:25.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:03:25.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:03:25.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:03:25.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:03:25.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:03:25.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:03:25.200 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:03:25.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:03:25.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:03:25.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:03:25.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:03:25.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:03:25.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:03:25.229 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:25.229+0000 7f0f10333640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:03:25.232 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:25.232+0000 7f0f10333640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:03:25.233 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:25.233+0000 7f0f10333640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:03:25.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:03:25.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:03:25.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:03:25.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:03:25.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:03:25.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:03:25.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:03:25.503 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:03:25.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:03:25.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:03:25.752 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:25.753+0000 7f0f10333640 -1 Falling back to public interface 2024-06-03T07:03:25.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:03:26.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:03:26.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:03:26.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:03:26.791 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:03:26.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:03:26.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:03:27.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:03:28.024 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:28.024+0000 7f0f10333640 -1 osd.0 46 log_to_monitors true 2024-06-03T07:03:28.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:03:28.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:03:28.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:03:28.091 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:03:28.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:03:28.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:03:28.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:03:29.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:03:29.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:03:29.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:03:29.408 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:03:29.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:03:29.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:03:29.860 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 51 up_thru 51 down_at 47 last_clean_interval [5,46) [v2:127.0.0.1:6802/2140562040,v1:127.0.0.1:6803/2140562040] [v2:127.0.0.1:6804/2140562040,v1:127.0.0.1:6805/2140562040] exists,up 8cc55336-1d3d-4454-b6f9-edab91bdf7f3 2024-06-03T07:03:29.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:03:29.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:03:29.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:03:29.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:03:29.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:03:29.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:03:29.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:03:29.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:03:29.861 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:03:29.861 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:03:29.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:03:29.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:03:29.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:03:29.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:03:29.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:03:29.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:03:29.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:03:29.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:03:29.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:03:30.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:03:30.250 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:03:30.250 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:03:30.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:03:30.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:30.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:03:30.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332098 2024-06-03T07:03:30.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332098 2024-06-03T07:03:30.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-219043332098' 2024-06-03T07:03:30.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:30.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:03:30.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626438 2024-06-03T07:03:30.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626438 2024-06-03T07:03:30.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-219043332098 1-180388626438' 2024-06-03T07:03:30.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:30.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:03:30.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-06-03T07:03:30.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-06-03T07:03:30.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-219043332098 1-180388626438 2-64424509456' 2024-06-03T07:03:30.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:30.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-219043332098 2024-06-03T07:03:30.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:30.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:03:30.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-219043332098 2024-06-03T07:03:30.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:30.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332098 2024-06-03T07:03:30.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 219043332098' 2024-06-03T07:03:30.577 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 219043332098 2024-06-03T07:03:30.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:03:30.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332098 -lt 219043332098 2024-06-03T07:03:30.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:30.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-180388626438 2024-06-03T07:03:30.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:30.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:03:30.888 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-180388626438 2024-06-03T07:03:30.888 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:30.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626438 2024-06-03T07:03:30.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 180388626438' 2024-06-03T07:03:30.890 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 180388626438 2024-06-03T07:03:30.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:03:31.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626437 -lt 180388626438 2024-06-03T07:03:31.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:03:32.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:03:32.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:03:32.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626437 -lt 180388626438 2024-06-03T07:03:32.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:03:33.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T07:03:33.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:03:33.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626438 -lt 180388626438 2024-06-03T07:03:33.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:33.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-06-03T07:03:33.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:33.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:03:33.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-06-03T07:03:33.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:33.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-06-03T07:03:33.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-06-03T07:03:33.789 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509456 2024-06-03T07:03:33.789 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:03:34.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509456 2024-06-03T07:03:34.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:03:34.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:03:34.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:03:34.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:03:34.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:03:34.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:03:34.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:03:34.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:03:34.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:03:34.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:03:34.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:03:34.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:03:34.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:03:34.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:03:34.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:03:35.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:03:35.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:03:35.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:03:35.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-03T07:03:35.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-06-03T07:03:35.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-06-03T07:03:35.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:03:35.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T07:03:35.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-06-03T07:03:35.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:03:35.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:03:35.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:03:35.530 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:03:35.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:03:35.940 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:03:35.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:03:35.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:03:35.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:03:36.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-06-03T07:03:36.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-03T07:03:36.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-03T07:03:36.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-06-03T07:03:36.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:03:36.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:03:36.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:03:36.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:03:36.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-06-03T07:03:36.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:03:36.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:03:36.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:03:36.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:03:36.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:03:36.235 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:03:36.235 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:03:36.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:03:36.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:03:36.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:03:36.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:03:36.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-06-03T07:03:36.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:03:36.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:03:36.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:03:36.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:03:36.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:03:36.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-06-03T07:03:38.070 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-06-03T07:03:38.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:03:38.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:03:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:03:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:03:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:03:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:03:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:03:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:03:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:03:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:03:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:03:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:03:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:03:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:03:38.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:03:38.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:03:38.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:03:38.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:03:38.588 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:03:38.588 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:03:38.588 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:03:38.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:03:38.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:03:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:03:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:03:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:03:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:03:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:03:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:03:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:03:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:03:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:03:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:03:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:03:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:03:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:03:38.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:03:38.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:03:38.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:03:38.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:03:38.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:03:38.620 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:38.620+0000 7fdaf4925640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:03:38.622 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:38.622+0000 7fdaf4925640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:03:38.624 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:38.624+0000 7fdaf4925640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:03:38.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:03:38.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:03:38.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:03:38.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:03:38.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:03:38.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:03:38.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:03:38.892 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:03:38.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:03:38.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:03:39.146 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:39.146+0000 7fdaf4925640 -1 Falling back to public interface 2024-06-03T07:03:39.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:03:40.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:03:40.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:03:40.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:03:40.180 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:03:40.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:03:40.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:03:40.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:03:41.470 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:41.470+0000 7fdaf4925640 -1 osd.1 55 log_to_monitors true 2024-06-03T07:03:41.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:03:41.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:03:41.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:03:41.481 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:03:41.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:03:41.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:03:41.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:03:42.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:03:42.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:03:42.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:03:42.799 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:03:42.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:03:42.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:03:43.104 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 60 up_thru 60 down_at 56 last_clean_interval [42,55) [v2:127.0.0.1:6810/615346751,v1:127.0.0.1:6811/615346751] [v2:127.0.0.1:6812/615346751,v1:127.0.0.1:6813/615346751] exists,up e0c162d7-c974-42b9-b532-ed08abf9157d 2024-06-03T07:03:43.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:03:43.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:03:43.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:03:43.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:03:43.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:03:43.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:03:43.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:03:43.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:03:43.105 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:03:43.105 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:03:43.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:03:43.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:03:43.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:03:43.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:03:43.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:03:43.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:03:43.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:03:43.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:03:43.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:03:43.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:03:43.525 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:03:43.525 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:03:43.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:03:43.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:43.525 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:03:43.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332102 2024-06-03T07:03:43.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332102 2024-06-03T07:03:43.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-219043332102' 2024-06-03T07:03:43.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:43.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:03:43.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037762 2024-06-03T07:03:43.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037762 2024-06-03T07:03:43.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-219043332102 1-257698037762' 2024-06-03T07:03:43.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:43.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:03:43.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-06-03T07:03:43.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-06-03T07:03:43.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-219043332102 1-257698037762 2-64424509460' 2024-06-03T07:03:43.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:43.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-219043332102 2024-06-03T07:03:43.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:43.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:03:43.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-219043332102 2024-06-03T07:03:43.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:43.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332102 2024-06-03T07:03:43.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 219043332102' 2024-06-03T07:03:43.899 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 219043332102 2024-06-03T07:03:43.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:03:44.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332100 -lt 219043332102 2024-06-03T07:03:44.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:03:45.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:03:45.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:03:45.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332102 -lt 219043332102 2024-06-03T07:03:45.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:45.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-257698037762 2024-06-03T07:03:45.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:45.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:03:45.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-257698037762 2024-06-03T07:03:45.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:45.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037762 2024-06-03T07:03:45.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 257698037762' 2024-06-03T07:03:45.501 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 257698037762 2024-06-03T07:03:45.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:03:45.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037762 -lt 257698037762 2024-06-03T07:03:45.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:45.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-06-03T07:03:45.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:45.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:03:45.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-06-03T07:03:45.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:45.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-06-03T07:03:45.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-06-03T07:03:45.794 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509460 2024-06-03T07:03:45.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:03:46.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-06-03T07:03:46.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:03:46.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:03:46.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:03:46.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:03:46.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:03:46.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:03:46.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:03:46.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:03:46.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:03:46.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:03:46.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:03:46.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:03:46.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:03:46.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:03:46.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:03:47.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:03:47.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:03:47.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:03:47.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-03T07:03:47.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-06-03T07:03:47.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-06-03T07:03:47.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:03:47.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T07:03:47.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-06-03T07:03:47.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:03:47.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:03:47.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:03:47.539 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:03:47.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:03:48.016 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:03:48.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:03:48.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:03:48.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:03:48.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-06-03T07:03:48.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-03T07:03:48.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-03T07:03:48.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-06-03T07:03:48.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-06-03T07:03:48.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-06-03T07:03:48.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-03T07:03:48.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:03:48.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:03:48.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:03:48.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:03:48.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-03T07:03:48.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:03:48.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:03:48.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:03:48.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:03:48.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:03:48.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:03:48.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:03:48.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:03:48.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:03:48.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:03:48.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:03:48.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-03T07:03:48.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:03:48.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:03:48.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:03:48.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:03:48.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:03:48.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-03T07:03:51.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:03:51.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:03:51.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:03:51.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:03:51.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:03:51.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:03:51.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:03:51.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:03:51.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:03:51.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:03:51.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:03:51.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:03:51.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:03:51.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:03:51.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:03:51.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:03:51.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:03:51.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:03:51.301 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:03:51.301 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:03:51.301 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:03:51.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:03:51.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:03:51.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:03:51.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:03:51.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:03:51.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:03:51.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:03:51.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:03:51.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:03:51.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:03:51.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:03:51.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:03:51.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:03:51.303 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:03:51.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:03:51.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:03:51.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:03:51.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:03:51.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:03:51.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:03:51.332 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:51.332+0000 7fabc121a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:03:51.334 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:51.334+0000 7fabc121a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:03:51.335 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:51.336+0000 7fabc121a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:03:51.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:03:51.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:03:51.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:03:51.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:03:51.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:03:51.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:03:51.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:03:51.632 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:03:51.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:03:51.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:03:51.854 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:51.855+0000 7fabc121a640 -1 Falling back to public interface 2024-06-03T07:03:51.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:03:52.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:03:52.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:03:52.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:03:52.962 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:03:52.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:03:52.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:03:53.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:03:54.157 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:03:54.156+0000 7fabc121a640 -1 osd.0 64 log_to_monitors true 2024-06-03T07:03:54.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:03:54.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:03:54.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:03:54.261 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:03:54.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:03:54.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:03:54.606 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 69 up_thru 60 down_at 65 last_clean_interval [51,64) [v2:127.0.0.1:6802/3785909039,v1:127.0.0.1:6803/3785909039] [v2:127.0.0.1:6804/3785909039,v1:127.0.0.1:6805/3785909039] exists,up 8cc55336-1d3d-4454-b6f9-edab91bdf7f3 2024-06-03T07:03:54.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:03:54.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:03:54.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:03:54.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:03:54.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:03:54.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:03:54.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:03:54.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:03:54.608 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:03:54.608 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:03:54.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:03:54.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:03:54.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:03:54.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:03:54.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:03:54.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:03:54.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:03:54.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:03:54.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:03:55.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:03:55.250 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:03:55.250 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:03:55.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:03:55.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:55.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:03:55.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743426 2024-06-03T07:03:55.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743426 2024-06-03T07:03:55.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743426' 2024-06-03T07:03:55.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:55.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:03:55.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037765 2024-06-03T07:03:55.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037765 2024-06-03T07:03:55.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743426 1-257698037765' 2024-06-03T07:03:55.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:03:55.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:03:55.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-06-03T07:03:55.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-06-03T07:03:55.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743426 1-257698037765 2-64424509463' 2024-06-03T07:03:55.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:55.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-296352743426 2024-06-03T07:03:55.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:55.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:03:55.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-296352743426 2024-06-03T07:03:55.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:55.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743426 2024-06-03T07:03:55.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 296352743426' 2024-06-03T07:03:55.757 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 296352743426 2024-06-03T07:03:55.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:03:56.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 296352743426 2024-06-03T07:03:56.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:03:57.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:03:57.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:03:57.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743426 -lt 296352743426 2024-06-03T07:03:57.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:57.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-257698037765 2024-06-03T07:03:57.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:57.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:03:57.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-257698037765 2024-06-03T07:03:57.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:57.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037765 2024-06-03T07:03:57.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 257698037765' 2024-06-03T07:03:57.353 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 257698037765 2024-06-03T07:03:57.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:03:57.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037765 -lt 257698037765 2024-06-03T07:03:57.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:03:57.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-06-03T07:03:57.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:03:57.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:03:57.648 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-06-03T07:03:57.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:03:57.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-06-03T07:03:57.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-06-03T07:03:57.650 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 64424509463 2024-06-03T07:03:57.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:03:57.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509463 2024-06-03T07:03:57.943 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:03:57.943 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:03:57.944 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:03:58.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:03:58.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:03:58.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:03:58.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:03:58.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:03:58.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:03:58.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:03:58.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:03:58.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T07:03:58.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:03:58.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:03:58.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:03:58.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-03T07:03:58.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-06-03T07:03:58.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T07:03:58.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-06-03T07:03:58.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:03:58.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T07:03:59.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:03:59.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:03:59.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:03:59.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:03:59.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:03:59.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:03:59.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:03:59.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:03:59.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T07:03:59.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:03:59.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:03:59.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:03:59.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-03T07:03:59.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-06-03T07:03:59.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T07:03:59.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T07:03:59.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T07:03:59.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T07:03:59.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T07:03:59.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T07:03:59.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T07:04:00.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T07:04:00.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T07:04:00.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T07:04:00.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:04:00.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T07:04:00.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:04:00.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:04:00.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:04:00.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:04:00.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:04:00.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:04:00.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:04:00.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:04:00.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:04:00.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:04:00.552 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:04:00.552 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:04:00.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:04:00.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:04:00.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:04:00.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-03T07:04:00.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:04:00.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:04:00.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-03T07:04:00.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:04:00.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-03T07:04:00.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:04:00.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:04:00.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-03T07:04:00.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:04:00.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-03T07:04:00.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:04:00.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:04:00.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:04:00.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:04:00.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:04:01.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:04:01.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-03T07:04:01.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:04:01.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:04:01.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-03T07:04:01.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:04:01.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-03T07:04:01.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-03T07:04:03.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-03T07:04:03.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:04:03.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:04:03.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-03T07:04:03.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:04:03.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T07:04:03.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:04:03.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:04:03.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:04:03.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:04:03.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T07:04:03.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T07:04:03.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:04:03.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:04:03.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:04:03.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:04:03.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:04:03.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:04:03.362 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:04:03.362 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:04:03.362 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:04:03.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:04:03.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:04:03.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:04:03.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:04:03.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:04:03.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:04:03.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:04:03.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:04:03.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:04:03.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:04:03.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:04:03.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T07:04:03.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-03T07:04:03.364 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.2 2024-06-03T07:04:03.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:04:03.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-03T07:04:03.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-03T07:04:03.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:04:03.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:04:03.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:04:03.393 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:03.394+0000 7f60c6230640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:04:03.419 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:03.419+0000 7f60c6230640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:04:03.420 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:03.420+0000 7f60c6230640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:04:03.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-03T07:04:03.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:04:03.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T07:04:03.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:04:03.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:04:03.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:03.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:04:03.689 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:04:03.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:03.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:04:03.938 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:03.938+0000 7f60c6230640 -1 Falling back to public interface 2024-06-03T07:04:03.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:04:04.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:04:04.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:04.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:04:04.978 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:04:04.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:04.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:04:05.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:04:06.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:04:06.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:06.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:04:06.280 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:04:06.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:06.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:04:06.326 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:06.327+0000 7f60c6230640 -1 osd.2 71 log_to_monitors true 2024-06-03T07:04:06.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:04:07.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:04:07.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:07.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:04:07.580 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:04:07.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:07.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:04:08.023 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 up in weight 1 up_from 76 up_thru 76 down_at 72 last_clean_interval [15,71) [v2:127.0.0.1:6818/2213344426,v1:127.0.0.1:6819/2213344426] [v2:127.0.0.1:6820/2213344426,v1:127.0.0.1:6821/2213344426] exists,up 2dd04ed2-911c-48e6-9324-0875996ba9ca 2024-06-03T07:04:08.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:04:08.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:04:08.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:04:08.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:04:08.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:04:08.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:04:08.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:04:08.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:04:08.025 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:04:08.025 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:04:08.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:04:08.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:04:08.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:04:08.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:04:08.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:04:08.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:04:08.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:04:08.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:04:08.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:04:08.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:04:08.409 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:04:08.409 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:04:08.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:04:08.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:04:08.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:04:08.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743429 2024-06-03T07:04:08.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743429 2024-06-03T07:04:08.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743429' 2024-06-03T07:04:08.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:04:08.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:04:08.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037769 2024-06-03T07:04:08.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037769 2024-06-03T07:04:08.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743429 1-257698037769' 2024-06-03T07:04:08.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:04:08.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:04:08.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514498 2024-06-03T07:04:08.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514498 2024-06-03T07:04:08.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743429 1-257698037769 2-326417514498' 2024-06-03T07:04:08.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:04:08.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-296352743429 2024-06-03T07:04:08.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:04:08.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:04:08.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-296352743429 2024-06-03T07:04:08.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:04:08.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743429 2024-06-03T07:04:08.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 296352743429' 2024-06-03T07:04:08.750 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 296352743429 2024-06-03T07:04:08.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:04:09.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743428 -lt 296352743429 2024-06-03T07:04:09.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:04:10.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:04:10.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:04:10.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743428 -lt 296352743429 2024-06-03T07:04:10.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:04:11.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T07:04:11.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:04:11.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743430 -lt 296352743429 2024-06-03T07:04:11.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:04:11.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-257698037769 2024-06-03T07:04:11.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:04:11.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:04:11.691 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-257698037769 2024-06-03T07:04:11.691 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:04:11.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037769 2024-06-03T07:04:11.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 257698037769' 2024-06-03T07:04:11.692 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 257698037769 2024-06-03T07:04:11.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:04:12.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037769 -lt 257698037769 2024-06-03T07:04:12.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:04:12.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514498 2024-06-03T07:04:12.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:04:12.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:04:12.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514498 2024-06-03T07:04:12.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:04:12.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514498 2024-06-03T07:04:12.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514498' 2024-06-03T07:04:12.025 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 326417514498 2024-06-03T07:04:12.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:04:12.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514498 -lt 326417514498 2024-06-03T07:04:12.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:04:12.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:04:12.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:04:12.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:04:12.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:04:12.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:04:12.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:04:12.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:04:12.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:04:12.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:04:12.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:04:13.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:04:13.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:04:13.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:04:13.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:04:13.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:04:13.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:04:13.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:04:13.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-06-03T07:04:13.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-03T07:04:13.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:04:13.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:04:13.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-03T07:04:13.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:04:13.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-03T07:04:13.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:04:13.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:04:13.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-03T07:04:13.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:04:13.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-03T07:04:13.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:04:13.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:04:13.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:04:13.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:04:13.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:04:13.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:04:13.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-03T07:04:13.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:04:13.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:04:13.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-03T07:04:13.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:04:13.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-03T07:04:13.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-03T07:04:15.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-03T07:04:15.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:04:15.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:04:15.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-03T07:04:15.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:04:15.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T07:04:15.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:04:15.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:04:15.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:04:15.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:04:15.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T07:04:15.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T07:04:15.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:04:15.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:04:15.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:04:15.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:04:15.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:04:15.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:04:15.647 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:04:15.647 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:04:15.647 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:04:15.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:04:15.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:04:15.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:04:15.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:04:15.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:04:15.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:04:15.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:04:15.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:04:15.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:04:15.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:04:15.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:04:15.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T07:04:15.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-03T07:04:15.650 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.2 2024-06-03T07:04:15.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:04:15.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-03T07:04:15.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-03T07:04:15.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:04:15.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:04:15.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:04:15.678 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:15.678+0000 7f11c056a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:04:15.680 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:15.681+0000 7f11c056a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:04:15.682 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:15.682+0000 7f11c056a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:04:15.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-03T07:04:15.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:04:15.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T07:04:15.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:04:15.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:04:15.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:15.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:04:15.952 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:04:15.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:15.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:04:16.197 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:16.198+0000 7f11c056a640 -1 Falling back to public interface 2024-06-03T07:04:16.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:04:17.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:04:17.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:17.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:04:17.244 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:04:17.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:17.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:04:17.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:04:18.488 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:18.489+0000 7f11c056a640 -1 osd.2 78 log_to_monitors true 2024-06-03T07:04:18.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:04:18.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:18.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:04:18.548 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:04:18.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:18.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:04:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 up in weight 1 up_from 83 up_thru 76 down_at 79 last_clean_interval [76,78) [v2:127.0.0.1:6818/972542600,v1:127.0.0.1:6819/972542600] [v2:127.0.0.1:6820/972542600,v1:127.0.0.1:6821/972542600] exists,up 2dd04ed2-911c-48e6-9324-0875996ba9ca 2024-06-03T07:04:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:04:18.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:04:18.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:04:18.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:04:18.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:04:18.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:04:18.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:04:18.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:04:18.903 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:04:18.903 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:04:18.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:04:18.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:04:18.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:04:18.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:04:18.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:04:18.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:04:19.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:04:19.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:04:19.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:04:19.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:04:19.323 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:04:19.323 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:04:19.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:04:19.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:04:19.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:04:19.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743433 2024-06-03T07:04:19.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743433 2024-06-03T07:04:19.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743433' 2024-06-03T07:04:19.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:04:19.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:04:19.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037772 2024-06-03T07:04:19.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037772 2024-06-03T07:04:19.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743433 1-257698037772' 2024-06-03T07:04:19.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:04:19.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:04:19.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285570 2024-06-03T07:04:19.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285570 2024-06-03T07:04:19.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743433 1-257698037772 2-356482285570' 2024-06-03T07:04:19.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:04:19.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-296352743433 2024-06-03T07:04:19.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:04:19.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:04:19.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-296352743433 2024-06-03T07:04:19.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:04:19.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743433 2024-06-03T07:04:19.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 296352743433' 2024-06-03T07:04:19.749 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 296352743433 2024-06-03T07:04:19.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:04:20.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743431 -lt 296352743433 2024-06-03T07:04:20.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:04:21.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:04:21.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:04:21.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743433 -lt 296352743433 2024-06-03T07:04:21.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:04:21.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-257698037772 2024-06-03T07:04:21.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:04:21.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:04:21.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-257698037772 2024-06-03T07:04:21.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:04:21.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037772 2024-06-03T07:04:21.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 257698037772' 2024-06-03T07:04:21.451 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 257698037772 2024-06-03T07:04:21.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:04:21.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037772 -lt 257698037772 2024-06-03T07:04:21.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:04:21.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285570 2024-06-03T07:04:21.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:04:21.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:04:21.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285570 2024-06-03T07:04:21.743 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:04:21.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285570 2024-06-03T07:04:21.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285570' 2024-06-03T07:04:21.744 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 356482285570 2024-06-03T07:04:21.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:04:22.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285570 -lt 356482285570 2024-06-03T07:04:22.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:04:22.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:04:22.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:04:22.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:04:22.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:04:22.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:04:22.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:04:22.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:04:22.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:04:22.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:04:22.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:04:22.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:04:22.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:04:22.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:04:22.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:04:23.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:04:23.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-03T07:04:23.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T07:04:23.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-03T07:04:23.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:04:23.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T07:04:23.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:04:23.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:04:23.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:04:23.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:04:23.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:04:23.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:04:23.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:04:23.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:04:23.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:04:23.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:04:23.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:04:23.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:04:23.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:04:23.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T07:04:23.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T07:04:23.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T07:04:23.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T07:04:23.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T07:04:23.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T07:04:23.906 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T07:04:23.906 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T07:04:24.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T07:04:24.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T07:04:24.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T07:04:24.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:04:24.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T07:04:24.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:04:24.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:04:24.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:04:24.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:04:24.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:04:24.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:04:24.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:04:24.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:04:24.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:04:24.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:04:24.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:04:24.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:04:25.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:04:25.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:04:25.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:04:25.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-06-03T07:04:25.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-03T07:04:25.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-06-03T07:04:25.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-06-03T07:04:25.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:04:25.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T07:04:25.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-06-03T07:04:25.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:04:25.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:04:25.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:04:25.622 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:04:25.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:04:26.023 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:04:26.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:04:26.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:04:26.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:04:26.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-06-03T07:04:26.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-03T07:04:26.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-03T07:04:26.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-06-03T07:04:26.257 INFO:tasks.workunit.client.0.smithi039.stderr:2+0 records in 2024-06-03T07:04:26.257 INFO:tasks.workunit.client.0.smithi039.stderr:2+0 records out 2024-06-03T07:04:26.257 INFO:tasks.workunit.client.0.smithi039.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 7.1858e-05 s, 57.0 MB/s 2024-06-03T07:04:26.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:04:26.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:04:26.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:04:26.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:04:26.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:04:26.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:04:26.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:04:26.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:04:26.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:04:26.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:04:26.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:04:26.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:04:26.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:04:26.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:04:26.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:04:26.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:04:26.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:04:26.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:04:26.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:04:26.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:04:26.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:04:26.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:04:26.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:04:26.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:04:28.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:04:28.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:04:28.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:04:28.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:04:28.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:04:28.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:04:28.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:04:28.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:04:28.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:04:28.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:04:28.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:04:28.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:04:28.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:04:28.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:04:28.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:04:28.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:04:28.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:04:28.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:04:28.669 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:04:28.670 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:04:28.670 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:04:28.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:04:28.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:04:28.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:04:28.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:04:28.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:04:28.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:04:28.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:04:28.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:04:28.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:04:28.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:04:28.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:04:28.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:04:28.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:04:28.672 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:04:28.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:04:28.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:04:28.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:04:28.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:04:28.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:04:28.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:04:28.700 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:28.700+0000 7fe46380e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:04:28.702 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:28.702+0000 7fe46380e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:04:28.703 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:28.703+0000 7fe46380e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:04:28.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:04:28.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:04:28.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:04:28.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:04:28.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:04:28.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:28.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:04:28.973 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:04:28.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:28.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:04:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:29.219+0000 7fe46380e640 -1 Falling back to public interface 2024-06-03T07:04:29.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:04:30.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:04:30.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:30.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:04:30.266 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:04:30.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:30.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:04:30.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:04:31.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:04:31.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:31.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:04:31.572 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:04:31.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:31.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:04:31.612 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:31.612+0000 7fe46380e640 -1 osd.1 87 log_to_monitors true 2024-06-03T07:04:31.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:04:32.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:04:32.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:32.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:04:32.880 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:04:32.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:32.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:04:33.264 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 92 up_thru 92 down_at 88 last_clean_interval [60,87) [v2:127.0.0.1:6810/1100593716,v1:127.0.0.1:6811/1100593716] [v2:127.0.0.1:6812/1100593716,v1:127.0.0.1:6813/1100593716] exists,up e0c162d7-c974-42b9-b532-ed08abf9157d 2024-06-03T07:04:33.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:04:33.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:04:33.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:04:33.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:04:33.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:04:33.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:04:33.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:04:33.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:04:33.266 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:04:33.266 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:04:33.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:04:33.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:04:33.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:04:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:04:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:04:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:04:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:04:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:04:33.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:04:33.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:04:33.700 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:04:33.700 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:04:33.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:04:33.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:04:33.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:04:33.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743436 2024-06-03T07:04:33.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743436 2024-06-03T07:04:33.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743436' 2024-06-03T07:04:33.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:04:33.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:04:33.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991234 2024-06-03T07:04:33.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991234 2024-06-03T07:04:33.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743436 1-395136991234' 2024-06-03T07:04:33.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:04:33.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:04:34.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285574 2024-06-03T07:04:34.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285574 2024-06-03T07:04:34.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743436 1-395136991234 2-356482285574' 2024-06-03T07:04:34.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:04:34.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-296352743436 2024-06-03T07:04:34.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:04:34.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:04:34.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-296352743436 2024-06-03T07:04:34.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:04:34.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743436 2024-06-03T07:04:34.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 296352743436' 2024-06-03T07:04:34.027 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 296352743436 2024-06-03T07:04:34.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:04:34.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743435 -lt 296352743436 2024-06-03T07:04:34.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:04:35.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:04:35.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:04:35.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743437 -lt 296352743436 2024-06-03T07:04:35.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:04:35.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-395136991234 2024-06-03T07:04:35.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:04:35.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:04:35.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-395136991234 2024-06-03T07:04:35.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:04:35.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991234 2024-06-03T07:04:35.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 395136991234' 2024-06-03T07:04:35.641 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 395136991234 2024-06-03T07:04:35.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:04:35.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991234 -lt 395136991234 2024-06-03T07:04:35.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:04:35.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285574 2024-06-03T07:04:35.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:04:35.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:04:35.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285574 2024-06-03T07:04:35.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:04:35.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285574 2024-06-03T07:04:35.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285574' 2024-06-03T07:04:35.937 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 356482285574 2024-06-03T07:04:35.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:04:36.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285574 -lt 356482285574 2024-06-03T07:04:36.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:04:36.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:04:36.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:04:36.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:04:36.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:04:36.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:04:36.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:04:36.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:04:36.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:04:36.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:04:36.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:04:36.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:04:36.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:04:36.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:04:36.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:04:37.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:04:37.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:04:37.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:04:37.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-03T07:04:37.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-06-03T07:04:37.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-06-03T07:04:37.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:04:37.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T07:04:37.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-06-03T07:04:37.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:04:37.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:04:37.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:04:37.765 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:04:37.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:04:38.174 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:04:38.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:04:38.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:04:38.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:04:38.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-06-03T07:04:38.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-03T07:04:38.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-03T07:04:38.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-03T07:04:38.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:04:38.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:04:38.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:04:38.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:04:38.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-03T07:04:38.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:04:38.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:04:38.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:04:38.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:04:38.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:04:38.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:04:38.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:04:38.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:04:38.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:04:38.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:04:38.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:04:38.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-03T07:04:38.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:04:38.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:04:38.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:04:38.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:04:38.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:04:38.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-06-03T07:04:40.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:04:40.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:04:40.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:04:40.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:04:40.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:04:40.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:04:40.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:04:40.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:04:40.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:04:40.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:04:40.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:04:40.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:04:40.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:04:40.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:04:40.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:04:40.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:04:40.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:04:40.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:04:40.832 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:04:40.832 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:04:40.832 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:04:40.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:04:40.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:04:40.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:04:40.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:04:40.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:04:40.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:04:40.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:04:40.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:04:40.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:04:40.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:04:40.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:04:40.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:04:40.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:04:40.834 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:04:40.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:04:40.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:04:40.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:04:40.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:04:40.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:04:40.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:04:40.863 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:40.863+0000 7f7c6df0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:04:40.865 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:40.866+0000 7f7c6df0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:04:40.867 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:40.867+0000 7f7c6df0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:04:41.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:04:41.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:04:41.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:04:41.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:04:41.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:04:41.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:41.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:04:41.133 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:04:41.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:41.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:04:41.389 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:41.389+0000 7f7c6df0c640 -1 Falling back to public interface 2024-06-03T07:04:41.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:04:42.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:04:42.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:42.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:04:42.505 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:04:42.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:42.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:04:42.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:04:43.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:04:43.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:43.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:04:43.833 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:04:43.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:43.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:04:43.913 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:43.912+0000 7f7c6df0c640 -1 osd.0 96 log_to_monitors true 2024-06-03T07:04:44.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:04:45.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:04:45.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:45.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:04:45.161 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:04:45.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:45.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:04:45.588 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 101 up_thru 101 down_at 97 last_clean_interval [69,96) [v2:127.0.0.1:6802/1329029039,v1:127.0.0.1:6803/1329029039] [v2:127.0.0.1:6804/1329029039,v1:127.0.0.1:6805/1329029039] exists,up 8cc55336-1d3d-4454-b6f9-edab91bdf7f3 2024-06-03T07:04:45.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:04:45.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:04:45.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:04:45.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:04:45.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:04:45.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:04:45.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:04:45.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:04:45.589 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:04:45.589 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:04:45.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:04:45.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:04:45.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:04:45.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:04:45.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:04:45.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:04:45.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:04:45.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:04:45.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:04:46.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:04:46.018 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:04:46.019 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:04:46.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:04:46.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:04:46.019 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:04:46.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696898 2024-06-03T07:04:46.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696898 2024-06-03T07:04:46.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696898' 2024-06-03T07:04:46.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:04:46.126 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:04:46.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991237 2024-06-03T07:04:46.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991237 2024-06-03T07:04:46.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696898 1-395136991237' 2024-06-03T07:04:46.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:04:46.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:04:46.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285577 2024-06-03T07:04:46.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285577 2024-06-03T07:04:46.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696898 1-395136991237 2-356482285577' 2024-06-03T07:04:46.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:04:46.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-433791696898 2024-06-03T07:04:46.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:04:46.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:04:46.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-433791696898 2024-06-03T07:04:46.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:04:46.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696898 2024-06-03T07:04:46.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 433791696898' 2024-06-03T07:04:46.345 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 433791696898 2024-06-03T07:04:46.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:04:46.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696898 -lt 433791696898 2024-06-03T07:04:46.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:04:46.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-395136991237 2024-06-03T07:04:46.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:04:46.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:04:46.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-395136991237 2024-06-03T07:04:46.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:04:46.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991237 2024-06-03T07:04:46.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 395136991237' 2024-06-03T07:04:46.641 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 395136991237 2024-06-03T07:04:46.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:04:46.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991237 -lt 395136991237 2024-06-03T07:04:46.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:04:46.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285577 2024-06-03T07:04:46.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:04:46.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:04:46.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285577 2024-06-03T07:04:46.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:04:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285577 2024-06-03T07:04:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285577' 2024-06-03T07:04:46.931 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 356482285577 2024-06-03T07:04:46.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:04:47.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285577 -lt 356482285577 2024-06-03T07:04:47.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:04:47.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:04:47.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:04:47.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:04:47.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:04:47.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:04:47.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:04:47.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:04:47.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:04:47.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:04:47.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:04:47.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:04:47.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:04:47.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:04:47.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:04:48.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:04:48.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:04:48.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:04:48.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-06-03T07:04:48.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-03T07:04:48.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:04:48.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:04:48.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:04:48.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:04:48.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-03T07:04:48.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:04:48.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:04:48.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:04:48.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:04:48.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:04:48.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:04:48.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:04:48.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:04:48.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:04:48.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:04:48.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:04:48.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-03T07:04:48.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:04:48.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:04:48.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:04:48.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:04:48.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:04:48.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-03T07:04:50.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:04:50.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:04:50.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:04:50.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:04:50.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:04:50.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:04:50.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:04:50.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:04:50.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:04:50.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:04:50.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:04:50.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:04:50.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:04:50.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:04:50.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:04:50.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:04:50.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:04:50.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:04:50.499 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:04:50.499 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:04:50.499 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:04:50.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:04:50.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:04:50.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:04:50.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:04:50.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:04:50.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:04:50.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:04:50.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:04:50.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:04:50.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:04:50.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:04:50.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:04:50.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:04:50.501 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:04:50.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:04:50.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:04:50.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:04:50.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:04:50.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:04:50.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:04:50.529 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:50.528+0000 7faca5296640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:04:50.542 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:50.541+0000 7faca5296640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:04:50.543 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:50.542+0000 7faca5296640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:04:50.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:04:50.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:04:50.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:04:50.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:04:50.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:04:50.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:50.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:04:50.801 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:04:50.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:50.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:04:51.057 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:51.057+0000 7faca5296640 -1 Falling back to public interface 2024-06-03T07:04:51.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:04:52.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:04:52.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:52.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:04:52.093 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:04:52.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:52.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:04:52.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:04:53.382 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:04:53.382+0000 7faca5296640 -1 osd.1 103 log_to_monitors true 2024-06-03T07:04:53.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:04:53.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:53.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:04:53.393 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:04:53.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:53.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:04:53.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:04:54.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:04:54.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:04:54.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:04:54.694 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:04:54.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:04:54.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:04:54.994 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 108 up_thru 108 down_at 104 last_clean_interval [92,103) [v2:127.0.0.1:6810/3896042077,v1:127.0.0.1:6811/3896042077] [v2:127.0.0.1:6812/3896042077,v1:127.0.0.1:6813/3896042077] exists,up e0c162d7-c974-42b9-b532-ed08abf9157d 2024-06-03T07:04:54.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:04:54.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:04:54.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:04:54.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:04:54.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:04:54.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:04:54.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:04:54.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:04:54.995 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:04:54.995 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:04:54.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:04:54.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:04:54.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:04:55.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:04:55.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:04:55.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:04:55.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:04:55.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:04:55.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:04:55.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:04:55.394 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:04:55.394 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:04:55.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:04:55.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:04:55.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:04:55.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696901 2024-06-03T07:04:55.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696901 2024-06-03T07:04:55.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696901' 2024-06-03T07:04:55.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:04:55.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:04:55.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467970 2024-06-03T07:04:55.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467970 2024-06-03T07:04:55.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696901 1-463856467970' 2024-06-03T07:04:55.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:04:55.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:04:55.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285580 2024-06-03T07:04:55.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285580 2024-06-03T07:04:55.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696901 1-463856467970 2-356482285580' 2024-06-03T07:04:55.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:04:55.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-433791696901 2024-06-03T07:04:55.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:04:55.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:04:55.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-433791696901 2024-06-03T07:04:55.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:04:55.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696901 2024-06-03T07:04:55.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 433791696901' 2024-06-03T07:04:55.752 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 433791696901 2024-06-03T07:04:55.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:04:56.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696900 -lt 433791696901 2024-06-03T07:04:56.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:04:57.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:04:57.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:04:57.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696901 -lt 433791696901 2024-06-03T07:04:57.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:04:57.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467970 2024-06-03T07:04:57.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:04:57.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:04:57.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467970 2024-06-03T07:04:57.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:04:57.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467970 2024-06-03T07:04:57.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467970' 2024-06-03T07:04:57.347 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 463856467970 2024-06-03T07:04:57.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:04:57.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467970 -lt 463856467970 2024-06-03T07:04:57.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:04:57.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285580 2024-06-03T07:04:57.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:04:57.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:04:57.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285580 2024-06-03T07:04:57.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:04:57.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285580 2024-06-03T07:04:57.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285580' 2024-06-03T07:04:57.683 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 356482285580 2024-06-03T07:04:57.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:04:57.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285580 -lt 356482285580 2024-06-03T07:04:57.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:04:57.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:04:57.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:04:58.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:04:58.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:04:58.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:04:58.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:04:58.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:04:58.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:04:58.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:04:58.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:04:58.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:04:58.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:04:58.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:04:58.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:04:58.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:04:58.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:04:58.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:04:58.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-03T07:04:58.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-06-03T07:04:58.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-06-03T07:04:58.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:04:58.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T07:04:58.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-06-03T07:04:58.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:04:58.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:04:58.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:04:59.365 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:04:59.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:04:59.868 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:04:59.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:04:59.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:04:59.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:05:00.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-06-03T07:05:00.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-03T07:05:00.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-03T07:05:00.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-06-03T07:05:00.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-06-03T07:05:00.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-06-03T07:05:00.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-03T07:05:00.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:05:00.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:05:00.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:05:00.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:05:00.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-03T07:05:00.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:05:00.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:05:00.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:05:00.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:05:00.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:05:00.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:05:00.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:05:00.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:05:00.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:05:00.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:05:00.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:05:00.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-03T07:05:00.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:05:00.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:05:00.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:05:00.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:05:00.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:05:00.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-06-03T07:05:02.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:05:02.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:05:02.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:05:02.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:05:02.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:05:02.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:05:02.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:05:02.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:05:02.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:05:02.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:05:02.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:05:02.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:05:02.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:05:02.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:05:02.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:05:02.235 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:05:02.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:05:02.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:05:02.236 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:05:02.236 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:05:02.236 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:05:02.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:05:02.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:05:02.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:05:02.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:05:02.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:05:02.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:05:02.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:05:02.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:05:02.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:05:02.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:05:02.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:05:02.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:05:02.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:05:02.238 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:05:02.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:05:02.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:05:02.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:05:02.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:05:02.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:05:02.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:05:02.268 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:02.268+0000 7fc882c43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:05:02.271 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:02.271+0000 7fc882c43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:05:02.273 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:02.273+0000 7fc882c43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:05:02.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:05:02.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:05:02.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:05:02.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:05:02.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:05:02.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:02.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:05:02.553 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:05:02.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:02.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:02.790 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:02.791+0000 7fc882c43640 -1 Falling back to public interface 2024-06-03T07:05:02.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:03.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:03.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:03.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:05:03.842 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:05:03.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:03.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:04.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:05.139 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:05.139+0000 7fc882c43640 -1 osd.0 112 log_to_monitors true 2024-06-03T07:05:05.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:05.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:05.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:05:05.146 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:05:05.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:05.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:05.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:06.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:06.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:06.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:05:06.497 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:05:06.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:06.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:06.844 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 117 up_thru 117 down_at 113 last_clean_interval [101,112) [v2:127.0.0.1:6802/3266820967,v1:127.0.0.1:6803/3266820967] [v2:127.0.0.1:6804/3266820967,v1:127.0.0.1:6805/3266820967] exists,up 8cc55336-1d3d-4454-b6f9-edab91bdf7f3 2024-06-03T07:05:06.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:05:06.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:05:06.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:05:06.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:05:06.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:05:06.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:05:06.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:05:06.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:05:06.845 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:05:06.845 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:05:06.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:05:06.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:05:06.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:05:06.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:05:06.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:05:06.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:05:06.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:05:06.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:05:06.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:05:07.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:05:07.229 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:05:07.229 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:05:07.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:05:07.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:05:07.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:05:07.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=502511173634 2024-06-03T07:05:07.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 502511173634 2024-06-03T07:05:07.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-502511173634' 2024-06-03T07:05:07.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:05:07.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:05:07.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467973 2024-06-03T07:05:07.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467973 2024-06-03T07:05:07.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-502511173634 1-463856467973' 2024-06-03T07:05:07.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:05:07.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:05:07.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285583 2024-06-03T07:05:07.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285583 2024-06-03T07:05:07.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-502511173634 1-463856467973 2-356482285583' 2024-06-03T07:05:07.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:05:07.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-502511173634 2024-06-03T07:05:07.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:05:07.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:05:07.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-502511173634 2024-06-03T07:05:07.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:05:07.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=502511173634 2024-06-03T07:05:07.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 502511173634' 2024-06-03T07:05:07.558 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 502511173634 2024-06-03T07:05:07.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:05:07.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 502511173634 2024-06-03T07:05:07.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:05:08.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:05:08.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:05:09.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 502511173634 -lt 502511173634 2024-06-03T07:05:09.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:05:09.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467973 2024-06-03T07:05:09.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:05:09.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:05:09.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467973 2024-06-03T07:05:09.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:05:09.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467973 2024-06-03T07:05:09.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467973' 2024-06-03T07:05:09.154 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 463856467973 2024-06-03T07:05:09.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:05:09.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467973 -lt 463856467973 2024-06-03T07:05:09.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:05:09.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285583 2024-06-03T07:05:09.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:05:09.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:05:09.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285583 2024-06-03T07:05:09.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:05:09.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285583 2024-06-03T07:05:09.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285583' 2024-06-03T07:05:09.512 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 356482285583 2024-06-03T07:05:09.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:05:09.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285583 -lt 356482285583 2024-06-03T07:05:09.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:05:09.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:05:09.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:05:10.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:05:10.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:05:10.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:05:10.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:05:10.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:05:10.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:05:10.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:05:10.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:05:10.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:05:10.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:05:10.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:05:10.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:05:10.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:05:10.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:05:10.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:05:10.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-03T07:05:10.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:05:10.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:05:10.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:05:10.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:05:10.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-03T07:05:10.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:05:10.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:05:10.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:05:10.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:05:10.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:05:10.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:05:10.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:05:10.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:05:10.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:05:10.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:05:11.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:05:11.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-03T07:05:11.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:05:11.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:05:11.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:05:11.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:05:11.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:05:11.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-03T07:05:13.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:05:13.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:05:13.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:05:13.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:05:13.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:05:13.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:05:13.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:05:13.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:05:13.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:05:13.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:05:13.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:05:13.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:05:13.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:05:13.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:05:13.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:05:13.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:05:13.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:05:13.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:05:13.442 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:05:13.442 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:05:13.442 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:05:13.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:05:13.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:05:13.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:05:13.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:05:13.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:05:13.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:05:13.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:05:13.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:05:13.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:05:13.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:05:13.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:05:13.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:05:13.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:05:13.444 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:05:13.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:05:13.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:05:13.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:05:13.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:05:13.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:05:13.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:05:13.473 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:13.474+0000 7f562fa11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:05:13.506 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:13.506+0000 7f562fa11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:05:13.507 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:13.507+0000 7f562fa11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:05:13.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:05:13.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:05:13.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:05:13.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:05:13.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:05:13.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:13.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:05:13.747 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:05:13.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:13.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:14.022 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:14.022+0000 7f562fa11640 -1 Falling back to public interface 2024-06-03T07:05:14.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:15.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:15.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:15.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:05:15.036 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:05:15.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:15.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:15.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:16.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:16.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:16.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:05:16.336 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:05:16.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:16.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:16.368 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:16.369+0000 7f562fa11640 -1 osd.0 119 log_to_monitors true 2024-06-03T07:05:16.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:17.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:17.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:17.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:05:17.749 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:05:17.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:17.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:18.143 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 124 up_thru 124 down_at 120 last_clean_interval [117,119) [v2:127.0.0.1:6802/2342845838,v1:127.0.0.1:6803/2342845838] [v2:127.0.0.1:6804/2342845838,v1:127.0.0.1:6805/2342845838] exists,up 8cc55336-1d3d-4454-b6f9-edab91bdf7f3 2024-06-03T07:05:18.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:05:18.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:05:18.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:05:18.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:05:18.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:05:18.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:05:18.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:05:18.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:05:18.144 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:05:18.144 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:05:18.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:05:18.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:05:18.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:05:18.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:05:18.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:05:18.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:05:18.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:05:18.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:05:18.240 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:05:18.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:05:18.566 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:05:18.566 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:05:18.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:05:18.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:05:18.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:05:18.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=532575944706 2024-06-03T07:05:18.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 532575944706 2024-06-03T07:05:18.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-532575944706' 2024-06-03T07:05:18.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:05:18.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:05:18.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467977 2024-06-03T07:05:18.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467977 2024-06-03T07:05:18.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-532575944706 1-463856467977' 2024-06-03T07:05:18.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:05:18.786 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:05:18.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285587 2024-06-03T07:05:18.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285587 2024-06-03T07:05:18.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-532575944706 1-463856467977 2-356482285587' 2024-06-03T07:05:18.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:05:18.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-532575944706 2024-06-03T07:05:18.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:05:18.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:05:18.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-532575944706 2024-06-03T07:05:18.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:05:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=532575944706 2024-06-03T07:05:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 532575944706' 2024-06-03T07:05:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 532575944706 2024-06-03T07:05:18.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:05:19.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 532575944706 2024-06-03T07:05:19.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:05:20.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:05:20.189 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:05:20.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 532575944706 2024-06-03T07:05:20.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:05:21.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T07:05:21.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:05:21.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 532575944706 -lt 532575944706 2024-06-03T07:05:21.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:05:21.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467977 2024-06-03T07:05:21.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:05:21.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:05:21.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467977 2024-06-03T07:05:21.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:05:21.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467977 2024-06-03T07:05:21.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467977' 2024-06-03T07:05:21.797 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 463856467977 2024-06-03T07:05:21.798 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:05:22.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467977 -lt 463856467977 2024-06-03T07:05:22.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:05:22.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285587 2024-06-03T07:05:22.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:05:22.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:05:22.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285587 2024-06-03T07:05:22.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:05:22.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285587 2024-06-03T07:05:22.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285587' 2024-06-03T07:05:22.091 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.2 seq 356482285587 2024-06-03T07:05:22.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:05:22.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285587 -lt 356482285587 2024-06-03T07:05:22.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:05:22.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:05:22.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:05:22.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:05:22.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:05:22.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:05:22.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:05:22.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:05:22.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:05:22.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:05:22.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:05:23.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:05:23.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:05:23.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:05:23.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:05:23.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:05:23.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:05:23.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:05:23.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-06-03T07:05:23.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-06-03T07:05:23.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T07:05:23.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-06-03T07:05:23.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-06-03T07:05:23.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T07:05:23.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-06-03T07:05:23.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-06-03T07:05:23.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-06-03T07:05:23.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-06-03T07:05:23.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:05:23.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:05:23.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:05:23.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:05:24.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-06-03T07:02:58.018083+0000 2024-06-03T07:05:24.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-06-03T07:05:24.466 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-06-03T07:05:24.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-06-03T07:02:58.018083+0000 2024-06-03T07:05:24.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-03T07:05:24.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T07:02:58.018083+0000 2024-06-03T07:05:24.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T07:05:24.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:05:24.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:05:24.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:05:24.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:05:24.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:05:24.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:05:24.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:05:24.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:02:58.018083+0000 '>' 2024-06-03T07:02:58.018083+0000 2024-06-03T07:05:24.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:05:25.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:05:25.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:05:25.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:05:25.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:05:25.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:05:25.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:05:25.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:05:26.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:02:58.018083+0000 '>' 2024-06-03T07:02:58.018083+0000 2024-06-03T07:05:26.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:05:27.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:05:27.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:05:27.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:05:27.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:05:27.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:05:27.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:05:27.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:05:27.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:02:58.018083+0000 '>' 2024-06-03T07:02:58.018083+0000 2024-06-03T07:05:27.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:05:28.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:05:28.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:05:28.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:05:28.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:05:28.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:05:28.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:05:28.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:05:28.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:02:58.018083+0000 '>' 2024-06-03T07:02:58.018083+0000 2024-06-03T07:05:28.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:05:29.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:05:29.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:05:29.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:05:29.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:05:29.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:05:29.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:05:29.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:05:30.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:02:58.018083+0000 '>' 2024-06-03T07:02:58.018083+0000 2024-06-03T07:05:30.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:05:31.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:05:31.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:05:31.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:05:31.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:05:31.019 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:05:31.019 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:05:31.019 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:05:31.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:05:25.413731+0000 '>' 2024-06-03T07:02:58.018083+0000 2024-06-03T07:05:31.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:05:31.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-06-03T07:05:31.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-06-03T07:05:31.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-06-03T07:05:31.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-03T07:05:31.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-06-03T07:05:31.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-06-03T07:05:31.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-06-03T07:05:31.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=124 2024-06-03T07:05:31.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-03T07:05:31.378 INFO:tasks.workunit.client.0.smithi039.stderr: . as $in 2024-06-03T07:05:31.378 INFO:tasks.workunit.client.0.smithi039.stderr: | if type == "object" then 2024-06-03T07:05:31.378 INFO:tasks.workunit.client.0.smithi039.stderr: reduce keys[] as $key 2024-06-03T07:05:31.378 INFO:tasks.workunit.client.0.smithi039.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-03T07:05:31.378 INFO:tasks.workunit.client.0.smithi039.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-03T07:05:31.378 INFO:tasks.workunit.client.0.smithi039.stderr: else f 2024-06-03T07:05:31.378 INFO:tasks.workunit.client.0.smithi039.stderr: end; 2024-06-03T07:05:31.379 INFO:tasks.workunit.client.0.smithi039.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-03T07:05:31.379 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-03T07:05:31.379 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-03T07:05:31.379 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-03T07:05:31.379 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-03T07:05:31.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-06-03T07:05:31.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-03T07:05:31.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-03T07:05:31.402 INFO:tasks.workunit.client.0.smithi039.stderr: . as $in 2024-06-03T07:05:31.402 INFO:tasks.workunit.client.0.smithi039.stderr: | if type == "object" then 2024-06-03T07:05:31.402 INFO:tasks.workunit.client.0.smithi039.stderr: reduce keys[] as $key 2024-06-03T07:05:31.402 INFO:tasks.workunit.client.0.smithi039.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-03T07:05:31.402 INFO:tasks.workunit.client.0.smithi039.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-03T07:05:31.402 INFO:tasks.workunit.client.0.smithi039.stderr: else f 2024-06-03T07:05:31.402 INFO:tasks.workunit.client.0.smithi039.stderr: end; 2024-06-03T07:05:31.402 INFO:tasks.workunit.client.0.smithi039.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-03T07:05:31.403 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-03T07:05:31.403 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-03T07:05:31.403 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-03T07:05:31.403 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-03T07:05:31.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-06-03T07:05:31.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-03T07:05:31.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-03T07:05:31.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-03T07:05:31.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-06-03T07:05:31.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-06-03T07:05:31.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-06-03T07:05:31.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-06-03T07:05:31.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:05:31.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:05:31.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:05:31.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:05:31.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:05:31.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-06-03T07:02:58.018083+0000 2024-06-03T07:05:31.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-06-03T07:05:31.983 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-06-03T07:05:31.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-06-03T07:02:58.018083+0000 last_deep_scrub_stamp 2024-06-03T07:05:31.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-03T07:05:31.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T07:02:58.018083+0000 2024-06-03T07:05:31.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-06-03T07:05:31.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:05:31.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:05:31.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:05:31.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:05:31.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:05:31.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:05:31.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:05:32.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:02:58.018083+0000 '>' 2024-06-03T07:02:58.018083+0000 2024-06-03T07:05:32.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:05:33.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:05:33.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:05:33.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:05:33.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:05:33.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:05:33.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:05:33.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:05:33.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:02:58.018083+0000 '>' 2024-06-03T07:02:58.018083+0000 2024-06-03T07:05:33.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:05:34.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:05:34.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:05:34.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:05:34.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:05:34.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:05:34.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:05:34.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:05:34.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:05:32.500754+0000 '>' 2024-06-03T07:02:58.018083+0000 2024-06-03T07:05:34.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:05:34.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-06-03T07:05:34.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-06-03T07:05:34.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-06-03T07:05:34.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-03T07:05:34.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-06-03T07:05:34.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-06-03T07:05:34.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-06-03T07:05:34.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=124 2024-06-03T07:05:34.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-06-03T07:05:34.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-03T07:05:34.937 INFO:tasks.workunit.client.0.smithi039.stderr: . as $in 2024-06-03T07:05:34.937 INFO:tasks.workunit.client.0.smithi039.stderr: | if type == "object" then 2024-06-03T07:05:34.937 INFO:tasks.workunit.client.0.smithi039.stderr: reduce keys[] as $key 2024-06-03T07:05:34.937 INFO:tasks.workunit.client.0.smithi039.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-03T07:05:34.938 INFO:tasks.workunit.client.0.smithi039.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-03T07:05:34.938 INFO:tasks.workunit.client.0.smithi039.stderr: else f 2024-06-03T07:05:34.938 INFO:tasks.workunit.client.0.smithi039.stderr: end; 2024-06-03T07:05:34.938 INFO:tasks.workunit.client.0.smithi039.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-03T07:05:34.938 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-03T07:05:34.938 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-03T07:05:34.938 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-03T07:05:34.938 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-03T07:05:34.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-06-03T07:05:34.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-03T07:05:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-03T07:05:34.961 INFO:tasks.workunit.client.0.smithi039.stderr: . as $in 2024-06-03T07:05:34.961 INFO:tasks.workunit.client.0.smithi039.stderr: | if type == "object" then 2024-06-03T07:05:34.961 INFO:tasks.workunit.client.0.smithi039.stderr: reduce keys[] as $key 2024-06-03T07:05:34.962 INFO:tasks.workunit.client.0.smithi039.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-03T07:05:34.962 INFO:tasks.workunit.client.0.smithi039.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-03T07:05:34.962 INFO:tasks.workunit.client.0.smithi039.stderr: else f 2024-06-03T07:05:34.962 INFO:tasks.workunit.client.0.smithi039.stderr: end; 2024-06-03T07:05:34.962 INFO:tasks.workunit.client.0.smithi039.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-03T07:05:34.962 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-03T07:05:34.962 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-03T07:05:34.962 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-03T07:05:34.962 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-03T07:05:34.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-06-03T07:05:34.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-03T07:05:34.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-03T07:05:34.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-03T07:05:34.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-06-03T07:05:34.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-06-03T07:05:34.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-06-03T07:05:35.376 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'ecpool' removed 2024-06-03T07:05:35.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T07:05:35.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:05:35.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:05:35.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:05:35.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:05:35.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:05:35.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:05:35.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:05:35.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:05:35.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:05:35.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:05:35.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:05:35.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:05:35.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:05:35.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:05:35.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:05:35.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:05:35.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:05:35.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:05:35.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:05:35.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:05:35.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:05:35.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:05:35.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:05:35.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:05:35.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:05:35.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:05:35.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:05:35.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:05:35.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T07:05:35.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T07:05:35.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T07:05:35.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T07:05:35.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:05:35.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:05:35.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:05:35.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:05:35.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:05:35.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:05:35.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:05:35.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:05:35.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:05:35.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:05:35.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:05:35.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:05:35.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:05:35.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:05:35.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:05:35.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:05:35.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:05:35.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:05:35.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:05:35.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:05:35.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:05:35.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:05:35.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:05:35.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:05:35.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:05:35.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:05:35.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:05:35.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:05:35.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T07:05:35.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T07:05:35.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:05:35.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:05:35.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T07:05:35.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T07:05:35.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T07:05:35.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T07:05:35.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T07:05:35.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-06-03T07:05:35.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-06-03T07:05:35.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-06-03T07:05:35.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-06-03T07:05:35.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-06-03T07:05:35.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T07:05:35.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T07:05:35.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T07:05:35.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T07:05:35.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T07:05:35.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-06-03T07:05:35.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T07:05:35.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:05:35.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:05:35.927 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:05:35.927 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:05:35.927 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:05:35.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:05:35.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-03T07:05:36.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T07:05:36.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T07:05:36.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:05:36.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:05:36.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T07:05:36.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T07:05:36.112 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:05:36.112 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:05:36.112 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:05:36.113 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:05:36.113 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:05:36.113 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:05:36.113 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:05:36.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:05:36.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T07:05:36.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T07:05:36.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:05:36.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:05:36.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T07:05:36.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T07:05:36.196 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:05:36.196 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:05:36.196 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:05:36.197 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:05:36.197 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:05:36.197 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:05:36.197 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:05:36.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:05:36.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T07:05:36.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-06-03T07:05:36.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T07:05:36.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T07:05:36.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T07:05:36.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T07:05:36.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T07:05:36.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T07:05:37.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T07:05:37.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:05:37.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:05:37.081 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:05:37.081 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:05:37.081 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:05:37.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:05:37.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:05:37.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:05:37.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-06-03T07:05:37.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:05:37.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:05:37.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T07:05:37.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:05:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:05:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:05:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:05:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:05:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:05:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:05:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:05:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:05:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:05:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:05:37.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:05:37.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:05:37.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:05:37.115 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:05:37.115 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:05:37.115 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:05:37.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:05:37.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:05:37.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:05:37.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:05:37.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:05:37.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:05:37.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:05:37.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:05:37.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:05:37.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:05:37.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:05:37.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:05:37.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T07:05:37.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:05:37.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:05:37.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3962cf66-f5f4-41c7-9b87-00f90eef0e73 2024-06-03T07:05:37.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3962cf66-f5f4-41c7-9b87-00f90eef0e73' 2024-06-03T07:05:37.117 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 3962cf66-f5f4-41c7-9b87-00f90eef0e73 2024-06-03T07:05:37.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:05:37.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBa11m71LtBxAAF4QXbM7g4uhVCr4cmJ5zBw== 2024-06-03T07:05:37.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBa11m71LtBxAAF4QXbM7g4uhVCr4cmJ5zBw=="}' 2024-06-03T07:05:37.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3962cf66-f5f4-41c7-9b87-00f90eef0e73 -i td/osd-scrub-repair/0/new.json 2024-06-03T07:05:37.369 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:05:37.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T07:05:37.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBa11m71LtBxAAF4QXbM7g4uhVCr4cmJ5zBw== --osd-uuid 3962cf66-f5f4-41c7-9b87-00f90eef0e73 2024-06-03T07:05:37.409 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:37.408+0000 7f1a53225640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:05:37.420 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:37.419+0000 7f1a53225640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:05:37.421 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:37.420+0000 7f1a53225640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:05:37.421 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:37.421+0000 7f1a53225640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:05:37.421 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:37.421+0000 7f1a53225640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:05:37.422 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:37.421+0000 7f1a53225640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:05:37.512 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:37.511+0000 7f1a53225640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T07:05:43.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T07:05:43.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:05:43.113 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T07:05:43.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T07:05:43.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:05:43.493 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T07:05:43.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T07:05:43.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:05:43.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:05:43.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:05:43.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:05:43.520 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:43.520+0000 7fb9137dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:05:43.522 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:43.521+0000 7fb9137dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:05:43.523 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:43.523+0000 7fb9137dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:05:43.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T07:05:43.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:05:43.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:05:43.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:05:43.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:05:43.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:43.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:05:43.785 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:05:43.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:43.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:44.043 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:44.043+0000 7fb9137dc640 -1 Falling back to public interface 2024-06-03T07:05:44.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:45.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:45.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:45.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:05:45.071 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:05:45.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:45.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:45.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:46.323 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:05:46.323+0000 7fb9137dc640 -1 osd.0 0 log_to_monitors true 2024-06-03T07:05:46.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:46.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:46.372 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:05:46.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:05:46.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:46.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:46.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:47.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:47.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:47.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:05:47.757 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:05:47.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:47.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:48.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:49.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:49.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:49.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:05:49.098 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:05:49.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:49.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:49.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:50.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:50.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:50.403 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:05:50.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:05:50.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:50.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:50.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:51.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:51.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:51.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:05:51.707 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:05:51.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:51.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:52.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:53.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:53.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:53.107 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:05:53.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:05:53.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:53.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:53.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:54.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:54.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:54.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:05:54.411 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:05:54.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:54.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:54.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:55.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:55.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:55.713 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:05:55.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:05:55.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:55.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:56.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:57.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:57.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:57.078 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:05:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:05:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:57.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:58.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:58.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:58.377 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:05:58.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:05:58.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:58.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:05:58.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:05:59.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:05:59.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:05:59.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:05:59.963 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:05:59.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:05:59.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:06:00.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:01.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:01.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:01.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:06:01.468 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:06:01.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:01.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:06:02.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:03.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:03.110 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:06:03.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:03.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:06:03.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:03.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:06:03.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:04.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:04.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:04.444 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:06:04.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:06:04.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:04.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:06:05.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:06.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:06.110 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:06:06.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:06.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:06:06.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:06.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:06:06.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:07.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:07.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:07.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:06:07.810 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:06:07.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:07.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:06:08.179 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2851635411,v1:127.0.0.1:6803/2851635411] [v2:127.0.0.1:6804/2851635411,v1:127.0.0.1:6805/2851635411] exists,up 3962cf66-f5f4-41c7-9b87-00f90eef0e73 2024-06-03T07:06:08.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:06:08.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:06:08.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:06:08.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-06-03T07:06:08.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:06:08.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:06:08.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T07:06:08.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:06:08.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:06:08.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:06:08.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:06:08.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:06:08.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:06:08.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:06:08.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:06:08.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:06:08.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:06:08.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:06:08.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:06:08.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:06:08.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:06:08.184 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:06:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:06:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:06:08.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:06:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:06:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:06:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:06:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:06:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:06:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:06:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:06:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:06:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:06:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:06:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:06:08.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T07:06:08.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:06:08.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:06:08.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:06:08.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b4579ed5-c309-4787-9209-8f3757d4856d' 2024-06-03T07:06:08.187 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:06:08.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:06:08.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBga11mZDEWDBAAZfXLm2nws6Onworvk/xwKQ== 2024-06-03T07:06:08.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBga11mZDEWDBAAZfXLm2nws6Onworvk/xwKQ=="}' 2024-06-03T07:06:08.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b4579ed5-c309-4787-9209-8f3757d4856d -i td/osd-scrub-repair/1/new.json 2024-06-03T07:06:08.611 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:06:08.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T07:06:08.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBga11mZDEWDBAAZfXLm2nws6Onworvk/xwKQ== --osd-uuid b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:06:08.652 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:06:08.653+0000 7f587b0b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:06:08.695 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:06:08.696+0000 7f587b0b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:06:08.696 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:06:08.697+0000 7f587b0b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:06:08.697 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:06:08.697+0000 7f587b0b5640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:06:08.697 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:06:08.697+0000 7f587b0b5640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:06:08.697 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:06:08.697+0000 7f587b0b5640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:06:08.771 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:06:08.771+0000 7f587b0b5640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T07:06:14.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T07:06:14.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:06:14.697 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T07:06:14.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T07:06:14.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:06:15.077 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T07:06:15.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T07:06:15.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:06:15.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:06:15.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:06:15.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:06:15.105 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:06:15.106+0000 7fb4839a7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:06:15.107 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:06:15.108+0000 7fb4839a7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:06:15.108 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:06:15.109+0000 7fb4839a7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:06:15.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T07:06:15.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:06:15.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:06:15.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:06:15.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:06:15.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:15.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:06:15.369 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:06:15.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:15.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:15.627 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:06:15.628+0000 7fb4839a7640 -1 Falling back to public interface 2024-06-03T07:06:15.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:16.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:16.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:16.663 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:06:16.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:06:16.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:16.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:16.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:17.878 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:06:17.879+0000 7fb4839a7640 -1 osd.1 0 log_to_monitors true 2024-06-03T07:06:17.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:17.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:17.965 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:06:17.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:06:17.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:17.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:18.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:19.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:19.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:19.315 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:06:19.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:06:19.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:19.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:19.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:20.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:20.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:06:20.659 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:06:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:20.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:21.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:21.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:21.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:06:21.964 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:06:21.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:21.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:22.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:23.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:23.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:23.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:06:23.268 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:06:23.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:23.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:23.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:24.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:24.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:24.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:06:24.759 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:06:24.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:24.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:25.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:26.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:26.059 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:06:26.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:26.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:06:26.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:26.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:26.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:27.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:27.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:27.400 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:06:27.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:06:27.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:27.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:27.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:28.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:28.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:28.737 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:06:28.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:06:28.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:28.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:29.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:30.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:30.038 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:06:30.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:30.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:06:30.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:30.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:30.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:31.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:31.721 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:06:31.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:31.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:06:31.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:31.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:32.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:33.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:33.129 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:06:33.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:33.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:06:33.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:33.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:33.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:34.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:34.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:34.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:06:34.519 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:06:34.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:34.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:35.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:36.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:36.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:36.159 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:06:36.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:06:36.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:36.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:36.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:37.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:37.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:37.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:06:37.813 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:06:37.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:37.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:38.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:39.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:39.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:39.447 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:06:39.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:06:39.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:39.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:39.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:06:40.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:06:40.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:06:40.758 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T07:06:40.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T07:06:40.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:06:40.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:06:41.055 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2874143861,v1:127.0.0.1:6811/2874143861] [v2:127.0.0.1:6812/2874143861,v1:127.0.0.1:6813/2874143861] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:06:41.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:06:41.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:06:41.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:06:41.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-06-03T07:06:41.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T07:06:41.340 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-06-03T07:06:41.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T07:06:41.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T07:06:41.744 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T07:06:41.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:06:42.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T07:06:44.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-06-03T07:06:44.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:06:44.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:06:44.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:06:44.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:06:44.238 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:06:44.238 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:06:44.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:06:44.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:06:44.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:06:44.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:06:44.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:06:44.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:06:44.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:06:44.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:06:44.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:06:44.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:06:44.669 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:06:44.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:06:44.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:06:44.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:06:44.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-06-03T07:06:44.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-06-03T07:06:44.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-06-03T07:06:44.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:06:44.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:06:44.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705668 2024-06-03T07:06:44.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705668 2024-06-03T07:06:44.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-38654705668' 2024-06-03T07:06:44.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:06:44.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-06-03T07:06:44.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:06:44.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:06:44.888 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-06-03T07:06:44.888 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:06:44.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-06-03T07:06:44.889 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836493 2024-06-03T07:06:44.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-06-03T07:06:44.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:06:45.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836493 2024-06-03T07:06:45.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:06:46.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:06:46.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:06:46.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-06-03T07:06:46.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:06:46.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705668 2024-06-03T07:06:46.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:06:46.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:06:46.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705668 2024-06-03T07:06:46.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:06:46.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705668 2024-06-03T07:06:46.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2024-06-03T07:06:46.485 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 38654705668 2024-06-03T07:06:46.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:06:46.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705668 -lt 38654705668 2024-06-03T07:06:46.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:06:46.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:06:46.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:06:47.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T07:06:47.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:06:47.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:06:47.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:06:47.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:06:47.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:06:47.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:06:47.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:06:47.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T07:06:47.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:06:47.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:06:47.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:06:47.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T07:06:47.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:06:47.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:06:47.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-06-03T07:06:47.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-06-03T07:06:48.318 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' created 2024-06-03T07:06:48.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:06:49.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-06-03T07:06:49.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-06-03T07:06:49.826 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'csr_pool' created 2024-06-03T07:06:49.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:06:50.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-06-03T07:06:50.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:06:50.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:06:50.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:06:50.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:06:50.841 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:06:50.841 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:06:50.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:06:50.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:06:50.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:06:50.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:06:50.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:06:50.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:06:50.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:06:50.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:06:50.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:06:51.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:06:51.242 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:06:51.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:06:51.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:06:51.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:06:51.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-06-03T07:06:51.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-06-03T07:06:51.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-06-03T07:06:51.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:06:51.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:06:51.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705671 2024-06-03T07:06:51.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705671 2024-06-03T07:06:51.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-38654705671' 2024-06-03T07:06:51.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:06:51.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-06-03T07:06:51.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:06:51.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:06:51.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-06-03T07:06:51.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:06:51.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-06-03T07:06:51.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-06-03T07:06:51.460 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836495 2024-06-03T07:06:51.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:06:51.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-06-03T07:06:51.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:06:52.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:06:52.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:06:53.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-06-03T07:06:53.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:06:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T07:06:54.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:06:54.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-06-03T07:06:54.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:06:54.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705671 2024-06-03T07:06:54.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:06:54.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:06:54.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705671 2024-06-03T07:06:54.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:06:54.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705671 2024-06-03T07:06:54.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705671' 2024-06-03T07:06:54.360 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 38654705671 2024-06-03T07:06:54.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:06:54.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705671 -lt 38654705671 2024-06-03T07:06:54.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:06:54.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:06:54.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:06:55.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:06:55.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:06:55.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:06:55.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:06:55.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:06:55.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:06:55.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:06:55.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:06:55.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:06:55.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:06:55.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:06:55.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:06:55.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:06:55.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:06:55.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:06:55.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-06-03T07:06:55.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:06:55.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-06-03T07:06:55.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-06-03T07:06:55.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:06:55.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:06:55.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-06-03T07:06:55.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:06:55.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:06:55.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:06:56.164 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:06:56.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:06:56.674 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:06:56.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:06:56.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:06:56.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:06:56.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-06-03T07:06:57.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-06-03T07:06:57.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:06:57.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-06-03T07:06:57.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-06-03T07:06:57.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:06:57.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:06:57.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-06-03T07:06:57.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:06:57.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:06:57.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:06:57.553 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:06:57.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:06:58.081 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:06:58.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:06:58.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:06:58.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:06:58.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-06-03T07:06:58.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-06-03T07:06:58.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:06:58.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-06-03T07:06:58.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-06-03T07:06:58.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:06:58.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:06:58.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-06-03T07:06:58.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:06:58.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:06:58.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:06:58.917 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:06:58.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:06:59.446 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:06:59.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:06:59.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:06:59.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:06:59.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-06-03T07:06:59.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-06-03T07:06:59.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:06:59.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-06-03T07:06:59.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-06-03T07:06:59.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:06:59.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:06:59.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-06-03T07:06:59.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:06:59.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:06:59.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:00.317 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:00.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:00.844 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:00.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:00.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:00.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:01.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-06-03T07:07:01.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-06-03T07:07:01.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:01.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-06-03T07:07:01.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-06-03T07:07:01.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:07:01.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:07:01.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-06-03T07:07:01.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:07:01.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:07:01.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:01.648 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:01.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:02.175 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:02.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:02.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:02.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:02.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-06-03T07:07:02.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-06-03T07:07:02.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:02.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-06-03T07:07:02.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-06-03T07:07:02.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:07:02.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:07:02.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-06-03T07:07:02.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:07:02.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:07:02.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:03.037 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:03.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:03.464 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:03.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:03.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:03.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:03.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-06-03T07:07:03.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-06-03T07:07:03.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:03.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-06-03T07:07:03.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-06-03T07:07:03.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:07:03.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:07:03.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-06-03T07:07:03.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:07:03.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:07:03.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:04.493 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:04.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:04.904 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:04.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:04.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:04.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:05.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-06-03T07:07:05.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-06-03T07:07:05.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:05.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-06-03T07:07:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-06-03T07:07:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:07:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:07:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-06-03T07:07:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:07:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:07:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:05.832 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:05.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:06.268 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:06.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:06.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:06.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:06.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-06-03T07:07:06.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-06-03T07:07:06.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:06.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-06-03T07:07:06.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-06-03T07:07:06.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:07:06.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:07:06.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-06-03T07:07:06.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:07:06.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:07:06.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:07.180 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:07.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:07.700 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:07.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:07.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:07.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:07.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-06-03T07:07:08.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-06-03T07:07:08.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:08.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-06-03T07:07:08.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-06-03T07:07:08.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:07:08.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:07:08.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-06-03T07:07:08.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:07:08.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:07:08.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:08.545 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:08.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:09.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:09.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:09.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:09.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-06-03T07:07:09.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-06-03T07:07:09.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:09.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-06-03T07:07:09.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-06-03T07:07:09.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:07:09.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:07:09.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-06-03T07:07:09.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:07:09.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:07:09.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:09.893 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:09.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:10.453 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:10.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:10.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:10.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:10.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-06-03T07:07:10.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-06-03T07:07:10.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:10.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-06-03T07:07:10.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-06-03T07:07:10.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:07:10.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:07:10.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-06-03T07:07:10.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:07:10.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:07:10.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:11.399 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:11.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:11.801 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:11.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:11.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:11.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:12.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-06-03T07:07:12.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-06-03T07:07:12.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:12.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-06-03T07:07:12.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-06-03T07:07:12.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:07:12.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:07:12.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-06-03T07:07:12.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:07:12.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:07:12.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:12.680 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:12.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:13.192 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:13.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:13.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:13.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:13.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-06-03T07:07:13.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-06-03T07:07:13.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:13.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-06-03T07:07:13.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-06-03T07:07:13.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:07:13.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:07:13.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-06-03T07:07:13.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:07:13.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:07:13.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:14.037 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:14.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:14.565 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:14.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:14.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:14.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:14.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-06-03T07:07:14.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-06-03T07:07:14.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:14.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-06-03T07:07:14.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-06-03T07:07:14.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:07:14.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:07:14.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-06-03T07:07:14.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:07:14.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:07:14.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:15.392 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:15.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:15.786 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:15.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:15.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:15.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:16.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-06-03T07:07:16.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-06-03T07:07:16.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:16.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-06-03T07:07:16.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-06-03T07:07:16.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:07:16.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:07:16.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-06-03T07:07:16.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:07:16.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:07:16.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:16.724 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:16.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:17.251 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:17.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:17.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:17.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:17.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-06-03T07:07:17.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-06-03T07:07:17.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:17.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-06-03T07:07:17.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-06-03T07:07:17.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:07:17.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:07:17.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-06-03T07:07:17.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:07:17.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:07:17.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:18.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:18.640 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:18.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:18.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:18.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:18.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-06-03T07:07:19.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-06-03T07:07:19.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:19.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-06-03T07:07:19.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-06-03T07:07:19.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:07:19.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:07:19.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-06-03T07:07:19.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:07:19.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:07:19.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:19.611 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:19.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:20.063 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:20.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:20.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:20.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:20.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-06-03T07:07:20.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-06-03T07:07:20.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:20.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-06-03T07:07:20.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-06-03T07:07:20.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:07:20.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:07:20.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-06-03T07:07:20.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:07:20.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:07:20.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:07:20.933 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:07:20.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:07:21.344 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:07:21.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:07:21.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:07:21.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:07:21.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-06-03T07:07:21.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-06-03T07:07:21.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-06-03T07:07:21.878 INFO:tasks.workunit.client.0.smithi039.stderr:1025+0 records in 2024-06-03T07:07:21.878 INFO:tasks.workunit.client.0.smithi039.stderr:1025+0 records out 2024-06-03T07:07:21.878 INFO:tasks.workunit.client.0.smithi039.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00217848 s, 482 MB/s 2024-06-03T07:07:21.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-06-03T07:07:22.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-06-03T07:07:22.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-06-03T07:07:22.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-06-03T07:07:22.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-06-03T07:07:22.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-06-03T07:07:22.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T07:07:22.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-06-03T07:07:22.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-06-03T07:07:22.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-06-03T07:07:22.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-06-03T07:07:22.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-06-03T07:07:22.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T07:07:22.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-06-03T07:07:22.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-06-03T07:07:22.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T07:07:22.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T07:07:22.668 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:07:22.668 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:07:22.668 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:07:22.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.0.asok 2024-06-03T07:07:22.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-06-03T07:07:22.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.69989/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-06-03T07:07:22.750 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T07:07:22.750 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-06-03T07:07:22.750 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T07:07:22.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-06-03T07:07:22.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-03T07:07:22.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-03T07:07:22.759 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:07:22.759 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:07:22.759 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:07:22.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.1.asok 2024-06-03T07:07:22.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-06-03T07:07:22.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.69989/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-06-03T07:07:22.841 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T07:07:22.841 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-06-03T07:07:22.841 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T07:07:22.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-06-03T07:07:22.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-06-03T07:07:22.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:07:22.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:07:22.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:07:22.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:07:22.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:07:23.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-06-03T07:06:49.707580+0000 2024-06-03T07:07:23.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-06-03T07:07:23.405 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-06-03T07:07:23.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-06-03T07:06:49.707580+0000 last_deep_scrub_stamp 2024-06-03T07:07:23.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-03T07:07:23.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T07:06:49.707580+0000 2024-06-03T07:07:23.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-06-03T07:07:23.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:07:23.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:07:23.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:07:23.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:07:23.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:07:23.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:07:23.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:07:23.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:06:49.707580+0000 '>' 2024-06-03T07:06:49.707580+0000 2024-06-03T07:07:23.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:07:24.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:07:24.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:07:24.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:07:24.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:07:24.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:07:24.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:07:24.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:07:24.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:06:49.707580+0000 '>' 2024-06-03T07:06:49.707580+0000 2024-06-03T07:07:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:07:25.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:07:25.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:07:25.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:07:25.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:07:25.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:07:25.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:07:25.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:07:26.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:06:49.707580+0000 '>' 2024-06-03T07:06:49.707580+0000 2024-06-03T07:07:26.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:07:27.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:07:27.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:07:27.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:07:27.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:07:27.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:07:27.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:07:27.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:07:27.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:07:24.364147+0000 '>' 2024-06-03T07:06:49.707580+0000 2024-06-03T07:07:27.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:07:27.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-06-03T07:07:27.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:27.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-06-03T07:07:27.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-06-03T07:07:27.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-03T07:07:27.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:07:27.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-06-03T07:07:27.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-06-03T07:07:27.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:07:27.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:07:27.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:07:27.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:07:27.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:07:27.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:07:27.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:07:27.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:07:27.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:07:27.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:07:27.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:07:27.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:07:27.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:07:27.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:07:27.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:07:27.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:07:27.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:07:27.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:07:27.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:07:27.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:07:27.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:07:27.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:07:27.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:07:27.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:07:29.122 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-06-03T07:07:29.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:07:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:07:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:07:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:07:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:07:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:07:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:07:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:07:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:07:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:07:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:07:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:07:29.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:07:29.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:07:29.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:07:29.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:07:29.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:07:29.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:07:29.680 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:07:29.680 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:07:29.680 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:07:29.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:07:29.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:07:29.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:07:29.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:07:29.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:07:29.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:07:29.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:07:29.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:07:29.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:07:29.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:07:29.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:07:29.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:07:29.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:07:29.682 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:07:29.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:07:29.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:07:29.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:07:29.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:07:29.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:07:29.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:07:29.710 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:07:29.710+0000 7fea92f61640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:07:29.713 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:07:29.713+0000 7fea92f61640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:07:29.714 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:07:29.715+0000 7fea92f61640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:07:29.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:07:29.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:07:29.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:07:29.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:07:29.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:07:29.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:07:29.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:07:29.980 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:07:29.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:07:29.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:07:30.235 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:07:30.236+0000 7fea92f61640 -1 Falling back to public interface 2024-06-03T07:07:30.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:07:31.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:07:31.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:07:31.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:07:31.270 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:07:31.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:07:31.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:07:31.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:07:32.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:07:32.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:07:32.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:07:32.577 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:07:32.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:07:32.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:07:32.715 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:07:32.715+0000 7fea92f61640 -1 osd.1 62 log_to_monitors true 2024-06-03T07:07:32.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:07:33.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:07:33.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:07:33.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:07:33.920 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:07:33.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:07:33.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:07:34.228 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 66 up_thru 66 down_at 63 last_clean_interval [9,62) [v2:127.0.0.1:6810/1500051722,v1:127.0.0.1:6811/1500051722] [v2:127.0.0.1:6812/1500051722,v1:127.0.0.1:6813/1500051722] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:07:34.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:07:34.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:07:34.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:07:34.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:07:34.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:07:34.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:07:34.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:07:34.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:07:34.229 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:07:34.229 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:07:34.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:07:34.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:07:34.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:07:34.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:07:34.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:07:34.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:07:34.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:07:34.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:07:34.325 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:07:34.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:07:34.612 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:07:34.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:07:34.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:07:34.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:07:34.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-06-03T07:07:34.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-06-03T07:07:34.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-06-03T07:07:34.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:07:34.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:07:34.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-06-03T07:07:34.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-06-03T07:07:34.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-283467841538' 2024-06-03T07:07:34.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:07:34.865 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-06-03T07:07:34.865 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:07:34.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:07:34.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-06-03T07:07:34.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:07:34.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-06-03T07:07:34.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-06-03T07:07:34.867 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836505 2024-06-03T07:07:34.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:07:35.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836505 2024-06-03T07:07:35.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:07:36.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:07:36.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:07:36.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-06-03T07:07:36.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:07:36.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841538 2024-06-03T07:07:36.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:07:36.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:07:36.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841538 2024-06-03T07:07:36.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:07:36.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-06-03T07:07:36.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841538' 2024-06-03T07:07:36.485 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 283467841538 2024-06-03T07:07:36.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:07:36.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841538 -lt 283467841538 2024-06-03T07:07:36.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:07:36.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:07:36.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:07:37.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:07:37.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:07:37.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:07:37.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:07:37.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:07:37.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:07:37.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:07:37.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:07:37.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:07:37.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:07:37.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:07:37.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:07:37.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:07:37.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:07:37.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:07:37.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:37.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-06-03T07:07:37.783 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-06-03T07:07:37.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-03T07:07:37.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:07:37.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-06-03T07:07:37.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-06-03T07:07:37.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:07:37.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:07:37.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:07:37.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:07:37.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:07:37.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:07:37.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:07:37.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:07:37.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:07:37.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:07:37.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:07:37.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:07:37.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:07:37.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:07:37.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:07:37.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:07:38.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:07:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:07:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:07:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:07:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:07:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:07:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:07:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:07:39.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:07:39.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:07:39.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:07:39.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:07:39.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:07:39.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:07:39.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:07:39.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:07:39.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:07:39.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:07:39.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:07:39.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:07:39.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:07:39.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:07:39.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:07:39.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:07:39.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:07:39.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:07:39.983 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:07:39.983 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:07:39.983 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:07:39.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:07:39.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:07:39.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:07:39.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:07:39.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:07:39.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:07:39.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:07:39.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:07:39.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:07:39.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:07:39.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:07:39.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:07:39.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:07:39.985 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:07:39.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:07:39.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:07:39.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:07:39.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:07:39.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:07:39.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:07:40.014 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:07:40.015+0000 7fbd76940640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:07:40.017 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:07:40.017+0000 7fbd76940640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:07:40.018 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:07:40.019+0000 7fbd76940640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:07:40.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:07:40.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:07:40.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:07:40.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:07:40.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:07:40.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:07:40.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:07:40.288 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:07:40.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:07:40.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:07:40.538 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:07:40.539+0000 7fbd76940640 -1 Falling back to public interface 2024-06-03T07:07:40.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:07:41.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:07:41.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:07:41.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:07:41.582 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:07:41.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:07:41.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:07:41.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:07:42.815 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:07:42.815+0000 7fbd76940640 -1 osd.0 68 log_to_monitors true 2024-06-03T07:07:42.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:07:42.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:07:42.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:07:42.883 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:07:42.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:07:42.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:07:43.270 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 72 up_thru 67 down_at 69 last_clean_interval [5,68) [v2:127.0.0.1:6802/2690856922,v1:127.0.0.1:6803/2690856922] [v2:127.0.0.1:6804/2690856922,v1:127.0.0.1:6805/2690856922] exists,up 3962cf66-f5f4-41c7-9b87-00f90eef0e73 2024-06-03T07:07:43.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:07:43.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:07:43.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:07:43.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:07:43.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:07:43.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:07:43.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:07:43.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:07:43.271 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:07:43.271 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:07:43.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:07:43.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:07:43.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:07:43.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:07:43.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:07:43.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:07:43.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:07:43.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:07:43.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:07:43.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:07:43.724 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:07:43.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:07:43.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:07:43.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:07:43.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645314 2024-06-03T07:07:43.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645314 2024-06-03T07:07:43.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645314' 2024-06-03T07:07:43.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:07:43.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:07:44.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841541 2024-06-03T07:07:44.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841541 2024-06-03T07:07:44.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645314 1-283467841541' 2024-06-03T07:07:44.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:07:44.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-309237645314 2024-06-03T07:07:44.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:07:44.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:07:44.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-309237645314 2024-06-03T07:07:44.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:07:44.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645314 2024-06-03T07:07:44.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 309237645314' 2024-06-03T07:07:44.008 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 309237645314 2024-06-03T07:07:44.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:07:44.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 309237645314 2024-06-03T07:07:44.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:07:45.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:07:45.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:07:45.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645314 -lt 309237645314 2024-06-03T07:07:45.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:07:45.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841541 2024-06-03T07:07:45.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:07:45.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:07:45.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841541 2024-06-03T07:07:45.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:07:45.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841541 2024-06-03T07:07:45.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841541' 2024-06-03T07:07:45.610 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 283467841541 2024-06-03T07:07:45.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:07:45.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841541 -lt 283467841541 2024-06-03T07:07:45.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:07:45.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:07:45.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:07:46.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:07:46.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:07:46.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:07:46.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:07:46.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:07:46.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:07:46.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:07:46.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:07:46.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:07:46.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:07:46.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:07:46.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:07:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:07:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-03T07:07:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T07:07:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-03T07:07:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:07:46.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T07:07:47.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:07:47.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:07:47.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:07:47.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:07:47.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:07:47.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:07:47.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:07:47.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:07:47.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:07:47.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:07:47.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:07:47.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:07:47.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:07:47.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T07:07:47.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T07:07:47.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T07:07:47.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T07:07:47.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T07:07:47.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T07:07:47.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T07:07:47.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T07:07:48.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T07:07:48.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T07:07:48.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T07:07:48.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:07:48.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T07:07:48.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:07:48.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:07:48.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:07:48.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:07:48.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:07:48.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:07:48.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:07:48.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:07:48.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:07:48.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:07:48.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:07:48.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:07:48.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:07:48.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T07:07:48.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T07:07:48.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T07:07:48.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T07:07:48.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T07:07:48.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T07:07:48.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T07:07:48.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T07:07:49.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T07:07:49.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T07:07:49.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-03T07:07:49.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:07:49.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-03T07:07:49.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:07:49.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:07:49.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:07:49.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:07:49.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:07:49.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:07:49.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:07:49.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:07:49.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:07:49.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:07:49.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:07:49.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:07:50.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:07:50.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:07:50.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:07:50.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:07:50.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-06-03T07:07:50.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-06-03T07:07:50.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-03T07:07:50.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:07:50.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-06-03T07:07:50.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:07:50.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:07:50.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:07:50.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:07:50.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-06-03T07:07:50.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:07:50.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:07:50.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:07:50.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:07:50.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:07:50.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:07:50.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:07:50.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:07:50.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:07:50.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:07:50.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:07:50.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-06-03T07:07:50.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:07:50.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:07:50.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:07:50.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:07:50.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:07:50.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-06-03T07:07:51.865 INFO:tasks.workunit.client.0.smithi039.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-06-03T07:07:52.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:07:52.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:07:52.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:07:52.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:07:52.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:07:52.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:07:52.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:07:52.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:07:52.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:07:52.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:07:52.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:07:52.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:07:52.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:07:52.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:07:52.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:07:52.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:07:52.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:07:52.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:07:52.384 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:07:52.385 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:07:52.385 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:07:52.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:07:52.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:07:52.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:07:52.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:07:52.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:07:52.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:07:52.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:07:52.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:07:52.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:07:52.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:07:52.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:07:52.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:07:52.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:07:52.387 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:07:52.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:07:52.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:07:52.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:07:52.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:07:52.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:07:52.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:07:52.416 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:07:52.416+0000 7f1d6cc32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:07:52.418 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:07:52.418+0000 7f1d6cc32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:07:52.419 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:07:52.420+0000 7f1d6cc32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:07:52.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:07:52.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:07:52.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:07:52.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:07:52.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:07:52.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:07:52.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:07:52.686 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:07:52.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:07:52.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:07:52.942 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:07:52.943+0000 7f1d6cc32640 -1 Falling back to public interface 2024-06-03T07:07:52.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:07:53.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:07:53.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:07:53.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:07:53.974 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:07:53.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:07:53.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:07:54.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:07:55.290 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:07:55.291+0000 7f1d6cc32640 -1 osd.1 73 log_to_monitors true 2024-06-03T07:07:55.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:07:55.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:07:55.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:07:55.291 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:07:55.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:07:55.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:07:55.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:07:56.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:07:56.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:07:56.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:07:56.629 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:07:56.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:07:56.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:07:56.893 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [66,73) [v2:127.0.0.1:6810/1841332021,v1:127.0.0.1:6811/1841332021] [v2:127.0.0.1:6812/1841332021,v1:127.0.0.1:6813/1841332021] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:07:56.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:07:56.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:07:56.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:07:56.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:07:56.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:07:56.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:07:56.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:07:56.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:07:56.894 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:07:56.895 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:07:56.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:07:56.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:07:56.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:07:56.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:07:56.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:07:56.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:07:56.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:07:56.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:07:56.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:07:57.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:07:57.279 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:07:57.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:07:57.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:07:57.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:07:57.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645317 2024-06-03T07:07:57.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645317 2024-06-03T07:07:57.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645317' 2024-06-03T07:07:57.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:07:57.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:07:57.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-06-03T07:07:57.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-06-03T07:07:57.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645317 1-330712481794' 2024-06-03T07:07:57.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:07:57.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-309237645317 2024-06-03T07:07:57.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:07:57.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:07:57.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-309237645317 2024-06-03T07:07:57.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:07:57.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645317 2024-06-03T07:07:57.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 309237645317' 2024-06-03T07:07:57.499 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 309237645317 2024-06-03T07:07:57.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:07:57.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645316 -lt 309237645317 2024-06-03T07:07:57.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:07:58.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:07:58.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:07:59.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645316 -lt 309237645317 2024-06-03T07:07:59.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:08:00.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T07:08:00.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:08:00.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645318 -lt 309237645317 2024-06-03T07:08:00.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:08:00.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481794 2024-06-03T07:08:00.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:08:00.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:08:00.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481794 2024-06-03T07:08:00.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:08:00.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-06-03T07:08:00.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-06-03T07:08:00.420 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 330712481794 2024-06-03T07:08:00.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:08:00.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481794 -lt 330712481794 2024-06-03T07:08:00.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:08:00.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:08:00.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:08:01.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:08:01.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:08:01.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:08:01.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:08:01.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:08:01.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:08:01.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:08:01.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:08:01.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:08:01.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:08:01.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:08:01.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:08:01.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:08:01.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:08:01.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:08:01.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:08:01.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-06-03T07:08:01.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-06-03T07:08:01.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-03T07:08:01.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:08:01.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-06-03T07:08:01.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:08:01.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:08:01.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:08:01.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:08:01.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-06-03T07:08:01.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:08:01.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:08:01.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:08:01.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:08:01.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:08:01.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:08:01.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:08:01.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:08:01.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:08:01.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:08:02.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:08:02.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-06-03T07:08:02.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:08:02.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:08:02.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:08:02.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:08:02.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:08:02.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-06-03T07:08:03.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:08:03.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:08:03.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:08:03.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:08:03.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:08:03.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:08:03.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:08:03.732 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:08:03.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:08:03.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:08:03.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:08:03.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:08:03.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:08:03.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:08:03.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:08:03.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:08:03.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:08:03.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:08:03.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:08:03.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:08:03.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:08:03.734 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:08:03.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:08:03.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:08:03.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:08:03.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:08:03.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:08:03.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:08:03.763 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:03.763+0000 7ff5cc226640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:03.766 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:03.766+0000 7ff5cc226640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:03.767 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:03.768+0000 7ff5cc226640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:04.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:08:04.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:08:04.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:08:04.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:08:04.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:08:04.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:04.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:08:04.041 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:08:04.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:04.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:08:04.283 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:04.284+0000 7ff5cc226640 -1 Falling back to public interface 2024-06-03T07:08:04.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:08:05.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:08:05.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:05.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:08:05.352 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:08:05.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:05.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:08:05.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:08:06.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:08:06.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:06.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:08:06.653 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:08:06.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:06.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:08:06.693 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:06.693+0000 7ff5cc226640 -1 osd.0 78 log_to_monitors true 2024-06-03T07:08:06.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:08:07.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:08:07.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:07.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:08:07.999 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:08:07.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:07.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:08:08.356 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [72,78) [v2:127.0.0.1:6802/148307464,v1:127.0.0.1:6803/148307464] [v2:127.0.0.1:6804/148307464,v1:127.0.0.1:6805/148307464] exists,up 3962cf66-f5f4-41c7-9b87-00f90eef0e73 2024-06-03T07:08:08.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:08:08.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:08:08.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:08:08.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:08:08.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:08:08.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:08:08.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:08:08.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:08:08.358 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:08:08.358 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:08:08.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:08:08.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:08:08.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:08:08.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:08:08.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:08:08.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:08:08.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:08:08.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:08:08.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:08:08.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:08:08.743 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:08:08.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:08:08.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:08:08.743 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:08:08.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-06-03T07:08:08.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-06-03T07:08:08.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318274' 2024-06-03T07:08:08.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:08:08.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:08:08.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481797 2024-06-03T07:08:08.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481797 2024-06-03T07:08:08.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318274 1-330712481797' 2024-06-03T07:08:08.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:08:08.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-352187318274 2024-06-03T07:08:08.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:08:08.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:08:08.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-352187318274 2024-06-03T07:08:08.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:08:08.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-06-03T07:08:08.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 352187318274' 2024-06-03T07:08:08.963 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 352187318274 2024-06-03T07:08:08.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:08:09.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 352187318274 2024-06-03T07:08:09.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:08:10.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:08:10.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:08:10.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-06-03T07:08:10.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:08:10.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481797 2024-06-03T07:08:10.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:08:10.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:08:10.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481797 2024-06-03T07:08:10.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:08:10.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481797 2024-06-03T07:08:10.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481797' 2024-06-03T07:08:10.563 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 330712481797 2024-06-03T07:08:10.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:08:10.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481797 -lt 330712481797 2024-06-03T07:08:10.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:08:10.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:08:10.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:08:11.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:08:11.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:08:11.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:08:11.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:08:11.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:08:11.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:08:11.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:08:11.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:08:11.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:08:11.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:08:11.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:08:11.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:08:11.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:08:11.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:08:11.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:08:11.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:08:11.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-06-03T07:08:11.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-06-03T07:08:11.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-03T07:08:11.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:08:11.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-06-03T07:08:11.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:08:11.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:08:11.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:08:11.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:08:11.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-06-03T07:08:11.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:08:11.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:08:11.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:08:11.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:08:11.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:08:11.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:08:11.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:08:11.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:08:11.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:08:11.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:08:12.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:08:12.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-06-03T07:08:12.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:08:12.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:08:12.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:08:12.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:08:12.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:08:12.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-06-03T07:08:13.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:08:13.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:08:13.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:08:13.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:08:13.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:08:13.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:08:13.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:08:13.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:08:13.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:08:13.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:08:13.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:08:13.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:08:13.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:08:13.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:08:13.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:08:13.819 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:08:13.819 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:08:13.819 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:08:13.820 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:08:13.820 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:08:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:08:13.821 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:08:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:08:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:08:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:08:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:08:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:08:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:08:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:08:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:08:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:08:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:08:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:08:13.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:08:13.822 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:08:13.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:08:13.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:08:13.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:08:13.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:08:13.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:08:13.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:08:13.851 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:13.851+0000 7f908be94640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:13.854 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:13.853+0000 7f908be94640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:13.855 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:13.854+0000 7f908be94640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:14.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:08:14.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:08:14.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:08:14.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:08:14.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:08:14.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:14.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:08:14.121 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:08:14.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:14.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:08:14.370 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:14.369+0000 7f908be94640 -1 Falling back to public interface 2024-06-03T07:08:14.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:08:15.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:08:15.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:15.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:08:15.414 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:08:15.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:15.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:08:15.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:08:16.709 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:16.708+0000 7f908be94640 -1 osd.1 84 log_to_monitors true 2024-06-03T07:08:16.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:08:16.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:16.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:08:16.714 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:08:16.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:16.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:08:17.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:08:18.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:08:18.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:18.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:08:18.198 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:08:18.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:18.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:08:18.495 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6810/3758083588,v1:127.0.0.1:6811/3758083588] [v2:127.0.0.1:6812/3758083588,v1:127.0.0.1:6813/3758083588] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:08:18.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:08:18.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:08:18.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:08:18.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:08:18.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:08:18.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:08:18.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:08:18.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:08:18.496 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:08:18.496 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:08:18.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:08:18.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:08:18.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:08:18.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:08:18.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:08:18.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:08:18.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:08:18.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:08:18.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:08:18.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:08:18.885 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:08:18.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:08:18.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:08:18.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:08:18.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318277 2024-06-03T07:08:18.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318277 2024-06-03T07:08:18.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318277' 2024-06-03T07:08:18.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:08:18.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:08:19.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-06-03T07:08:19.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-06-03T07:08:19.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318277 1-377957122050' 2024-06-03T07:08:19.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:08:19.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-352187318277 2024-06-03T07:08:19.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:08:19.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:08:19.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-352187318277 2024-06-03T07:08:19.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:08:19.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318277 2024-06-03T07:08:19.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 352187318277' 2024-06-03T07:08:19.104 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 352187318277 2024-06-03T07:08:19.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:08:19.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318276 -lt 352187318277 2024-06-03T07:08:19.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:08:20.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:08:20.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:08:20.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318277 -lt 352187318277 2024-06-03T07:08:20.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:08:20.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122050 2024-06-03T07:08:20.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:08:20.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:08:20.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122050 2024-06-03T07:08:20.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:08:20.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-06-03T07:08:20.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2024-06-03T07:08:20.856 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 377957122050 2024-06-03T07:08:20.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:08:21.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122050 -lt 377957122050 2024-06-03T07:08:21.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:08:21.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:08:21.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:08:21.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:08:21.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:08:21.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:08:21.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:08:21.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:08:21.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:08:21.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:08:21.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:08:21.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:08:21.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:08:21.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:08:21.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:08:22.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:08:22.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:08:22.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:08:22.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:08:22.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-06-03T07:08:22.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-06-03T07:08:22.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-03T07:08:22.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:08:22.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-06-03T07:08:22.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-06-03T07:08:22.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:08:22.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:08:22.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:08:22.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:08:22.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-06-03T07:08:22.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:08:22.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:08:22.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:08:22.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:08:22.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:08:22.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:08:22.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:08:22.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:08:22.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:08:22.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:08:22.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:08:22.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-06-03T07:08:22.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:08:22.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:08:22.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:08:22.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:08:22.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:08:22.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-06-03T07:08:24.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:08:24.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:08:24.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:08:24.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:08:24.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:08:24.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:08:24.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:08:24.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:08:24.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:08:24.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:08:24.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:08:24.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:08:24.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:08:24.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:08:24.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:08:24.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:08:24.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:08:24.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:08:24.136 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:08:24.136 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:08:24.136 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:08:24.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:08:24.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:08:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:08:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:08:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:08:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:08:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:08:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:08:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:08:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:08:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:08:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:08:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:08:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:08:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:08:24.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:08:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:08:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:08:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:08:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:08:24.167 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:24.168+0000 7fbb25b16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:24.170 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:24.171+0000 7fbb25b16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:24.171 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:24.172+0000 7fbb25b16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:24.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:08:24.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:08:24.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:08:24.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:08:24.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:08:24.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:24.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:08:24.439 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:08:24.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:24.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:08:24.694 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:24.695+0000 7fbb25b16640 -1 Falling back to public interface 2024-06-03T07:08:24.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:08:25.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:08:25.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:25.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:08:25.732 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:08:25.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:25.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:08:26.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:08:27.034 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:27.034+0000 7fbb25b16640 -1 osd.0 89 log_to_monitors true 2024-06-03T07:08:27.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:08:27.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:27.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:08:27.036 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:08:27.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:27.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:08:27.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:08:28.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:08:28.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:28.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:08:28.341 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:08:28.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:28.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:08:28.688 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 93 up_thru 93 down_at 90 last_clean_interval [82,89) [v2:127.0.0.1:6802/2814359939,v1:127.0.0.1:6803/2814359939] [v2:127.0.0.1:6804/2814359939,v1:127.0.0.1:6805/2814359939] exists,up 3962cf66-f5f4-41c7-9b87-00f90eef0e73 2024-06-03T07:08:28.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:08:28.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:08:28.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:08:28.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:08:28.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:08:28.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:08:28.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:08:28.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:08:28.690 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:08:28.690 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:08:28.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:08:28.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:08:28.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:08:28.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:08:28.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:08:28.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:08:28.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:08:28.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:08:28.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:08:29.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:08:29.074 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:08:29.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:08:29.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:08:29.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:08:29.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-06-03T07:08:29.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-06-03T07:08:29.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958530' 2024-06-03T07:08:29.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:08:29.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:08:29.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122053 2024-06-03T07:08:29.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122053 2024-06-03T07:08:29.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958530 1-377957122053' 2024-06-03T07:08:29.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:08:29.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-399431958530 2024-06-03T07:08:29.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:08:29.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:08:29.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-399431958530 2024-06-03T07:08:29.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:08:29.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-06-03T07:08:29.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 399431958530' 2024-06-03T07:08:29.294 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 399431958530 2024-06-03T07:08:29.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:08:29.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958530 -lt 399431958530 2024-06-03T07:08:29.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:08:29.582 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122053 2024-06-03T07:08:29.582 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:08:29.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:08:29.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122053 2024-06-03T07:08:29.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:08:29.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122053 2024-06-03T07:08:29.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122053' 2024-06-03T07:08:29.585 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 377957122053 2024-06-03T07:08:29.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:08:29.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122052 -lt 377957122053 2024-06-03T07:08:29.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:08:30.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:08:30.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:08:31.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122052 -lt 377957122053 2024-06-03T07:08:31.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:08:32.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T07:08:32.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:08:32.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122053 -lt 377957122053 2024-06-03T07:08:32.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:08:32.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:08:32.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:08:32.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:08:32.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:08:32.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:08:32.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:08:32.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:08:32.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:08:32.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:08:32.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:08:33.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:08:33.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:08:33.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:08:33.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:08:33.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:08:33.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:08:33.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:08:33.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-06-03T07:08:33.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:08:33.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-06-03T07:08:33.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-06-03T07:08:33.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-03T07:08:33.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:08:33.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-06-03T07:08:33.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-06-03T07:08:33.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:08:33.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:08:33.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:08:33.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:08:33.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-06-03T07:08:33.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:08:33.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:08:33.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:08:33.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:08:33.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:08:33.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:08:33.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:08:33.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:08:33.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:08:33.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:08:33.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:08:33.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-06-03T07:08:33.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:08:33.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:08:33.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:08:33.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:08:33.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:08:33.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-06-03T07:08:35.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:08:35.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:08:35.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:08:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:08:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:08:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:08:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:08:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:08:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:08:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:08:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:08:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:08:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:08:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:08:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:08:35.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:08:35.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:08:35.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:08:35.373 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:08:35.373 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:08:35.373 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:08:35.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:08:35.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:08:35.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:08:35.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:08:35.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:08:35.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:08:35.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:08:35.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:08:35.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:08:35.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:08:35.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:08:35.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:08:35.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:08:35.375 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:08:35.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:08:35.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:08:35.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:08:35.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:08:35.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:08:35.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:08:35.405 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:35.405+0000 7fc1c396e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:35.407 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:35.408+0000 7fc1c396e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:35.409 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:35.409+0000 7fc1c396e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:35.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:08:35.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:08:35.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:08:35.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:08:35.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:08:35.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:35.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:08:35.679 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:08:35.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:35.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:08:35.924 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:35.925+0000 7fc1c396e640 -1 Falling back to public interface 2024-06-03T07:08:35.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:08:37.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:08:37.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:37.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:08:37.001 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:08:37.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:37.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:08:37.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:08:38.268 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:38.269+0000 7fc1c396e640 -1 osd.1 95 log_to_monitors true 2024-06-03T07:08:38.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:08:38.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:38.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:08:38.304 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:08:38.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:38.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:08:38.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:08:39.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:08:39.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:39.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:08:39.607 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:08:39.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:39.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:08:39.911 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 99 up_thru 99 down_at 96 last_clean_interval [88,95) [v2:127.0.0.1:6810/4091800579,v1:127.0.0.1:6811/4091800579] [v2:127.0.0.1:6812/4091800579,v1:127.0.0.1:6813/4091800579] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:08:39.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:08:39.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:08:39.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:08:39.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:08:39.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:08:39.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:08:39.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:08:39.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:08:39.912 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:08:39.912 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:08:39.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:08:39.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:08:39.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:08:40.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:08:40.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:08:40.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:08:40.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:08:40.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:08:40.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:08:40.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:08:40.299 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:08:40.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:08:40.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:08:40.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:08:40.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958533 2024-06-03T07:08:40.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958533 2024-06-03T07:08:40.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958533' 2024-06-03T07:08:40.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:08:40.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:08:40.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-06-03T07:08:40.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-06-03T07:08:40.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958533 1-425201762306' 2024-06-03T07:08:40.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:08:40.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-399431958533 2024-06-03T07:08:40.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:08:40.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:08:40.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-399431958533 2024-06-03T07:08:40.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:08:40.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958533 2024-06-03T07:08:40.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 399431958533' 2024-06-03T07:08:40.518 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 399431958533 2024-06-03T07:08:40.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:08:40.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958532 -lt 399431958533 2024-06-03T07:08:40.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:08:41.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:08:41.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:08:42.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958533 -lt 399431958533 2024-06-03T07:08:42.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:08:42.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762306 2024-06-03T07:08:42.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:08:42.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:08:42.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762306 2024-06-03T07:08:42.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:08:42.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-06-03T07:08:42.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762306' 2024-06-03T07:08:42.111 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 425201762306 2024-06-03T07:08:42.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:08:42.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762306 -lt 425201762306 2024-06-03T07:08:42.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:08:42.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:08:42.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:08:42.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:08:42.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:08:42.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:08:42.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:08:42.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:08:42.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:08:42.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:08:42.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:08:43.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:08:43.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:08:43.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:08:43.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:08:43.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:08:43.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:08:43.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:08:43.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-06-03T07:08:43.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:08:43.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-06-03T07:08:43.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-06-03T07:08:43.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-03T07:08:43.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:08:43.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-06-03T07:08:43.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-06-03T07:08:43.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-06-03T07:08:43.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-06-03T07:08:43.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:08:43.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:08:43.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:08:43.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:08:43.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-06-03T07:08:43.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:08:43.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:08:43.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:08:43.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:08:43.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:08:43.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:08:43.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:08:43.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:08:43.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:08:43.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:08:43.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:08:43.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-06-03T07:08:43.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:08:43.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:08:43.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:08:43.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:08:43.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:08:43.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-06-03T07:08:45.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:08:45.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:08:45.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:08:45.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:08:45.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:08:45.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:08:45.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:08:45.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:08:45.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:08:45.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:08:45.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:08:45.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:08:45.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:08:45.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:08:45.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:08:45.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:08:45.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:08:45.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:08:45.375 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:08:45.375 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:08:45.375 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:08:45.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:08:45.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:08:45.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:08:45.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:08:45.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:08:45.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:08:45.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:08:45.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:08:45.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:08:45.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:08:45.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:08:45.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:08:45.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:08:45.377 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:08:45.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:08:45.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:08:45.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:08:45.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:08:45.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:08:45.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:08:45.406 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:45.407+0000 7fea5931c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:45.409 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:45.409+0000 7fea5931c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:45.410 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:45.411+0000 7fea5931c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:45.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:08:45.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:08:45.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:08:45.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:08:45.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:08:45.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:45.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:08:45.678 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:08:45.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:45.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:08:45.927 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:45.928+0000 7fea5931c640 -1 Falling back to public interface 2024-06-03T07:08:45.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:08:46.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:08:46.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:46.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:08:46.970 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:08:46.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:46.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:08:47.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:08:48.203 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:48.204+0000 7fea5931c640 -1 osd.0 101 log_to_monitors true 2024-06-03T07:08:48.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:08:48.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:48.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:08:48.271 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:08:48.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:48.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:08:48.692 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 105 up_thru 100 down_at 102 last_clean_interval [93,101) [v2:127.0.0.1:6802/1966219073,v1:127.0.0.1:6803/1966219073] [v2:127.0.0.1:6804/1966219073,v1:127.0.0.1:6805/1966219073] exists,up 3962cf66-f5f4-41c7-9b87-00f90eef0e73 2024-06-03T07:08:48.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:08:48.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:08:48.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:08:48.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:08:48.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:08:48.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:08:48.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:08:48.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:08:48.693 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:08:48.693 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:08:48.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:08:48.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:08:48.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:08:48.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:08:48.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:08:48.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:08:48.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:08:48.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:08:48.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:08:49.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:08:49.179 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:08:49.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:08:49.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:08:49.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:08:49.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-06-03T07:08:49.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-06-03T07:08:49.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566082' 2024-06-03T07:08:49.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:08:49.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:08:49.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762309 2024-06-03T07:08:49.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762309 2024-06-03T07:08:49.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566082 1-425201762309' 2024-06-03T07:08:49.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:08:49.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566082 2024-06-03T07:08:49.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:08:49.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:08:49.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566082 2024-06-03T07:08:49.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:08:49.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-06-03T07:08:49.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566082' 2024-06-03T07:08:49.481 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 450971566082 2024-06-03T07:08:49.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:08:49.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 450971566082 2024-06-03T07:08:49.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:08:50.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:08:50.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:08:51.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 450971566082 2024-06-03T07:08:51.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:08:52.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T07:08:52.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:08:52.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566082 -lt 450971566082 2024-06-03T07:08:52.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:08:52.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762309 2024-06-03T07:08:52.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:08:52.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:08:52.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762309 2024-06-03T07:08:52.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:08:52.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762309 2024-06-03T07:08:52.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762309' 2024-06-03T07:08:52.435 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 425201762309 2024-06-03T07:08:52.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:08:52.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762309 -lt 425201762309 2024-06-03T07:08:52.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:08:52.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:08:52.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:08:53.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:08:53.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:08:53.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:08:53.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:08:53.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:08:53.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:08:53.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:08:53.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:08:53.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:08:53.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:08:53.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:08:53.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:08:53.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:08:53.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-03T07:08:53.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T07:08:53.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-03T07:08:53.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:08:53.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T07:08:53.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:08:53.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:08:53.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:08:53.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:08:53.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:08:53.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:08:53.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:08:53.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:08:54.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:08:54.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:08:54.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:08:54.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:08:54.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:08:54.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T07:08:54.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T07:08:54.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T07:08:54.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T07:08:54.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T07:08:54.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T07:08:54.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T07:08:54.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T07:08:54.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T07:08:54.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T07:08:54.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T07:08:54.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:08:54.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T07:08:55.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:08:55.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:08:55.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:08:55.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:08:55.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:08:55.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:08:55.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:08:55.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:08:55.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:08:55.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:08:55.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:08:55.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:08:55.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:08:55.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:08:55.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:08:55.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-06-03T07:08:55.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:08:55.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:08:55.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:08:55.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:08:55.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-06-03T07:08:55.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:08:55.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:08:55.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:08:55.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:08:55.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:08:55.713 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:08:55.713 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:08:55.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:08:55.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:08:55.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:08:56.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:08:56.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-06-03T07:08:56.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:08:56.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:08:56.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:08:56.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:08:56.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:08:56.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-06-03T07:08:57.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:08:57.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:08:57.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:08:57.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:08:57.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:08:57.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:08:57.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:08:57.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:08:57.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:08:57.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:08:57.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:08:57.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:08:57.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:08:57.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:08:57.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:08:57.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:08:57.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:08:57.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:08:57.713 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:08:57.713 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:08:57.713 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:08:57.713 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:08:57.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:08:57.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:08:57.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:08:57.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:08:57.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:08:57.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:08:57.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:08:57.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:08:57.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:08:57.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:08:57.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:08:57.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:08:57.715 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:08:57.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:08:57.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:08:57.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:08:57.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:08:57.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:08:57.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:08:57.743 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:57.743+0000 7f0f26e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:57.745 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:57.746+0000 7f0f26e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:57.746 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:57.747+0000 7f0f26e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:08:58.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:08:58.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:08:58.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:08:58.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:08:58.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:08:58.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:58.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:08:58.071 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:08:58.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:58.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:08:58.266 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:08:58.267+0000 7f0f26e15640 -1 Falling back to public interface 2024-06-03T07:08:58.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:08:59.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:08:59.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:08:59.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:08:59.304 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:08:59.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:08:59.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:08:59.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:00.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:00.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:00.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:09:00.613 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:09:00.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:00.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:00.708 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:00.709+0000 7f0f26e15640 -1 osd.1 106 log_to_monitors true 2024-06-03T07:09:01.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:02.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:02.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:02.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:09:02.032 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:09:02.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:02.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:02.398 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 110 up_thru 110 down_at 107 last_clean_interval [99,106) [v2:127.0.0.1:6810/1956147386,v1:127.0.0.1:6811/1956147386] [v2:127.0.0.1:6812/1956147386,v1:127.0.0.1:6813/1956147386] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:09:02.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:09:02.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:09:02.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:09:02.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:09:02.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:09:02.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:09:02.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:09:02.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:09:02.399 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:09:02.399 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:09:02.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:09:02.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:09:02.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:09:02.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:09:02.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:09:02.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:09:02.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:09:02.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:09:02.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:09:02.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:09:02.783 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:09:02.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:09:02.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:09:02.783 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:09:02.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566085 2024-06-03T07:09:02.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566085 2024-06-03T07:09:02.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566085' 2024-06-03T07:09:02.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:09:02.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:09:02.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402562 2024-06-03T07:09:02.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402562 2024-06-03T07:09:02.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566085 1-472446402562' 2024-06-03T07:09:02.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:09:02.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566085 2024-06-03T07:09:02.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:09:03.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:09:03.001 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566085 2024-06-03T07:09:03.001 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:09:03.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566085 2024-06-03T07:09:03.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566085' 2024-06-03T07:09:03.002 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 450971566085 2024-06-03T07:09:03.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:09:03.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566084 -lt 450971566085 2024-06-03T07:09:03.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:09:04.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:09:04.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:09:04.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566085 -lt 450971566085 2024-06-03T07:09:04.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:09:04.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-472446402562 2024-06-03T07:09:04.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:09:04.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:09:04.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-472446402562 2024-06-03T07:09:04.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:09:04.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402562 2024-06-03T07:09:04.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 472446402562' 2024-06-03T07:09:04.595 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 472446402562 2024-06-03T07:09:04.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:09:04.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402562 -lt 472446402562 2024-06-03T07:09:04.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:09:04.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:09:04.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:09:05.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:09:05.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:09:05.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:09:05.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:09:05.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:09:05.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:09:05.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:09:05.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:09:05.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:09:05.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:09:05.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:09:05.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:09:05.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:09:05.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:09:05.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:09:05.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-06-03T07:09:05.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-06-03T07:09:05.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:09:05.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:09:05.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:09:05.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:09:05.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-06-03T07:09:05.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:09:05.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:09:05.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:09:05.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:09:05.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:09:05.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:09:05.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:09:05.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:09:05.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:09:05.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:09:06.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:09:06.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-06-03T07:09:06.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:09:06.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:09:06.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:09:06.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:09:06.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:09:06.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-06-03T07:09:07.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:09:07.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:09:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:09:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:09:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:09:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:09:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:09:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:09:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:09:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:09:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:09:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:09:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:09:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:09:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:09:07.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:09:07.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:09:07.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:09:07.934 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:09:07.934 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:09:07.934 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:09:07.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:09:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:09:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:09:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:09:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:09:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:09:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:09:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:09:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:09:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:09:07.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:09:07.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:09:07.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:09:07.937 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:09:07.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:09:07.937 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:09:07.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:09:07.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:09:07.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:09:07.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:09:07.966 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:07.965+0000 7f80d2223640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:07.968 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:07.967+0000 7f80d2223640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:07.969 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:07.969+0000 7f80d2223640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:08.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:09:08.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:09:08.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:09:08.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:09:08.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:09:08.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:08.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:09:08.236 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:09:08.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:08.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:08.483 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:08.483+0000 7f80d2223640 -1 Falling back to public interface 2024-06-03T07:09:08.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:09.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:09.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:09.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:09:09.523 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:09:09.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:09.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:09.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:10.814 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:10.813+0000 7f80d2223640 -1 osd.1 112 log_to_monitors true 2024-06-03T07:09:10.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:10.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:10.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:09:10.822 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:09:10.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:10.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:11.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:12.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:12.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:12.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:09:12.259 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:09:12.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:12.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:12.612 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 116 up_thru 116 down_at 113 last_clean_interval [110,112) [v2:127.0.0.1:6810/1536110488,v1:127.0.0.1:6811/1536110488] [v2:127.0.0.1:6812/1536110488,v1:127.0.0.1:6813/1536110488] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:09:12.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:09:12.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:09:12.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:09:12.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:09:12.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:09:12.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:09:12.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:09:12.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:09:12.614 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:09:12.614 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:09:12.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:09:12.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:09:12.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:09:12.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:09:12.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:09:12.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:09:12.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:09:12.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:09:12.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:09:13.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:09:13.003 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:09:13.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:09:13.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:09:13.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:09:13.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566088 2024-06-03T07:09:13.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566088 2024-06-03T07:09:13.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566088' 2024-06-03T07:09:13.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:09:13.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:09:13.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-06-03T07:09:13.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-06-03T07:09:13.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566088 1-498216206338' 2024-06-03T07:09:13.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:09:13.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566088 2024-06-03T07:09:13.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:09:13.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:09:13.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566088 2024-06-03T07:09:13.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:09:13.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566088 2024-06-03T07:09:13.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566088' 2024-06-03T07:09:13.222 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 450971566088 2024-06-03T07:09:13.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:09:13.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566088 -lt 450971566088 2024-06-03T07:09:13.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:09:13.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-498216206338 2024-06-03T07:09:13.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:09:13.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:09:13.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-498216206338 2024-06-03T07:09:13.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:09:13.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-06-03T07:09:13.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 498216206338' 2024-06-03T07:09:13.515 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 498216206338 2024-06-03T07:09:13.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:09:13.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206338 -lt 498216206338 2024-06-03T07:09:13.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:09:13.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:09:13.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:09:14.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:09:14.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:09:14.235 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:09:14.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:09:14.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:09:14.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:09:14.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:09:14.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:09:14.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:09:14.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:09:14.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:09:14.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:09:14.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:09:14.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:09:14.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:09:14.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-06-03T07:09:14.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:09:14.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-06-03T07:09:14.882 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-06-03T07:09:14.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-03T07:09:14.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:09:14.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-06-03T07:09:14.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:09:14.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:09:14.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:09:14.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:09:14.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-06-03T07:09:14.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:09:14.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:09:14.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:09:14.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:09:14.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:09:14.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:09:14.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:09:14.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:09:14.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:09:14.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:09:15.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:09:15.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-06-03T07:09:15.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:09:15.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:09:15.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:09:15.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:09:15.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:09:15.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-06-03T07:09:16.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:09:16.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:09:16.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:09:16.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:09:16.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:09:16.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:09:16.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:09:16.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:09:16.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:09:16.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:09:16.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:09:16.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:09:16.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:09:16.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:09:16.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:09:16.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:09:16.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:09:16.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:09:16.908 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:09:16.908 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:09:16.908 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:09:16.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:09:16.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:09:16.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:09:16.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:09:16.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:09:16.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:09:16.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:09:16.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:09:16.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:09:16.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:09:16.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:09:16.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:09:16.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:09:16.910 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:09:16.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:09:16.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:09:16.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:09:16.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:09:16.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:09:16.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:09:16.940 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:16.940+0000 7fb088013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:16.942 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:16.942+0000 7fb088013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:16.943 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:16.944+0000 7fb088013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:17.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:09:17.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:09:17.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:09:17.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:09:17.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:09:17.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:17.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:09:17.212 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:09:17.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:17.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:17.460 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:17.461+0000 7fb088013640 -1 Falling back to public interface 2024-06-03T07:09:17.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:18.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:18.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:18.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:09:18.578 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:09:18.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:18.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:18.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:19.851 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:19.851+0000 7fb088013640 -1 osd.1 117 log_to_monitors true 2024-06-03T07:09:19.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:19.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:19.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:09:19.873 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:09:19.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:19.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:20.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:21.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:21.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:21.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:09:21.186 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:09:21.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:21.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:21.488 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 121 up_thru 121 down_at 118 last_clean_interval [116,117) [v2:127.0.0.1:6810/60720828,v1:127.0.0.1:6811/60720828] [v2:127.0.0.1:6812/60720828,v1:127.0.0.1:6813/60720828] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:09:21.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:09:21.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:09:21.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:09:21.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:09:21.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:09:21.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:09:21.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:09:21.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:09:21.489 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:09:21.490 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:09:21.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:09:21.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:09:21.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:09:21.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:09:21.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:09:21.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:09:21.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:09:21.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:09:21.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:09:21.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:09:21.904 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:09:21.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:09:21.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:09:21.904 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:09:22.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566091 2024-06-03T07:09:22.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566091 2024-06-03T07:09:22.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566091' 2024-06-03T07:09:22.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:09:22.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:09:22.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=519691042818 2024-06-03T07:09:22.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 519691042818 2024-06-03T07:09:22.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566091 1-519691042818' 2024-06-03T07:09:22.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:09:22.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566091 2024-06-03T07:09:22.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:09:22.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:09:22.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566091 2024-06-03T07:09:22.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:09:22.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566091 2024-06-03T07:09:22.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566091' 2024-06-03T07:09:22.124 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 450971566091 2024-06-03T07:09:22.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:09:22.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566090 -lt 450971566091 2024-06-03T07:09:22.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:09:23.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:09:23.434 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:09:23.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566091 -lt 450971566091 2024-06-03T07:09:23.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:09:23.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-519691042818 2024-06-03T07:09:23.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:09:23.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:09:23.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-519691042818 2024-06-03T07:09:23.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:09:23.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=519691042818 2024-06-03T07:09:23.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 519691042818' 2024-06-03T07:09:23.741 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 519691042818 2024-06-03T07:09:23.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:09:24.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 519691042818 -lt 519691042818 2024-06-03T07:09:24.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:09:24.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:09:24.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:09:24.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:09:24.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:09:24.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:09:24.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:09:24.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:09:24.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:09:24.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:09:24.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:09:24.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:09:24.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:09:24.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:09:24.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:09:25.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:09:25.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:09:25.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:09:25.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-06-03T07:09:25.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-06-03T07:09:25.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-03T07:09:25.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:09:25.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:09:25.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:09:25.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:09:25.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-03T07:09:25.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:09:25.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:09:25.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:09:25.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:09:25.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:09:25.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:09:25.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:09:25.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:09:25.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:09:25.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:09:25.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:09:25.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-03T07:09:25.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:09:25.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:09:25.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:09:25.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:09:25.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:09:25.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-03T07:09:27.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:09:27.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:09:27.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:09:27.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:09:27.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:09:27.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:09:27.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:09:27.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:09:27.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:09:27.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:09:27.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:09:27.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:09:27.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:09:27.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:09:27.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:09:27.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:09:27.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:09:27.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:09:27.148 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:09:27.148 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:09:27.148 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:09:27.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:09:27.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:09:27.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:09:27.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:09:27.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:09:27.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:09:27.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:09:27.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:09:27.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:09:27.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:09:27.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:09:27.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:09:27.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:09:27.150 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:09:27.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:09:27.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:09:27.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:09:27.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:09:27.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:09:27.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:09:27.178 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:27.179+0000 7fdd0d243640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:27.181 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:27.181+0000 7fdd0d243640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:27.182 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:27.183+0000 7fdd0d243640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:27.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:09:27.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:09:27.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:09:27.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:09:27.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:09:27.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:27.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:09:27.450 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:09:27.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:27.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:27.698 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:27.698+0000 7fdd0d243640 -1 Falling back to public interface 2024-06-03T07:09:27.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:28.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:28.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:28.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:09:28.810 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:09:28.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:28.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:29.961 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:29.961+0000 7fdd0d243640 -1 osd.1 123 log_to_monitors true 2024-06-03T07:09:30.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:30.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:30.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:09:30.087 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:09:30.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:30.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:30.419 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 127 up_thru 121 down_at 124 last_clean_interval [121,123) [v2:127.0.0.1:6810/727477779,v1:127.0.0.1:6811/727477779] [v2:127.0.0.1:6812/727477779,v1:127.0.0.1:6813/727477779] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:09:30.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:09:30.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:09:30.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:09:30.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:09:30.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:09:30.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:09:30.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:09:30.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:09:30.420 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:09:30.420 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:09:30.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:09:30.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:09:30.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:09:30.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:09:30.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:09:30.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:09:30.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:09:30.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:09:30.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:09:30.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:09:30.901 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:09:30.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:09:30.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:09:30.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:09:31.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566094 2024-06-03T07:09:31.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566094 2024-06-03T07:09:31.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566094' 2024-06-03T07:09:31.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:09:31.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:09:31.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846594 2024-06-03T07:09:31.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846594 2024-06-03T07:09:31.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566094 1-545460846594' 2024-06-03T07:09:31.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:09:31.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566094 2024-06-03T07:09:31.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:09:31.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:09:31.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566094 2024-06-03T07:09:31.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:09:31.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566094 2024-06-03T07:09:31.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566094' 2024-06-03T07:09:31.263 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 450971566094 2024-06-03T07:09:31.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:09:31.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566094 -lt 450971566094 2024-06-03T07:09:31.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:09:31.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-545460846594 2024-06-03T07:09:31.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:09:31.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:09:31.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-545460846594 2024-06-03T07:09:31.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:09:31.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846594 2024-06-03T07:09:31.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 545460846594' 2024-06-03T07:09:31.654 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 545460846594 2024-06-03T07:09:31.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:09:31.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846594 -lt 545460846594 2024-06-03T07:09:31.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:09:31.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:09:31.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:09:32.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:09:32.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:09:32.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:09:32.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:09:32.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:09:32.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:09:32.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:09:32.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:09:32.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:09:32.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:09:32.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:09:32.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:09:32.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:09:32.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-03T07:09:32.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T07:09:32.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-03T07:09:32.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:09:32.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T07:09:33.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:09:33.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:09:33.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:09:33.084 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:09:33.084 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:09:33.084 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:09:33.084 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:09:33.084 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:09:33.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:09:33.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:09:33.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:09:33.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:09:33.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:09:33.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T07:09:33.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T07:09:33.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T07:09:33.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T07:09:33.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T07:09:33.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T07:09:33.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T07:09:33.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T07:09:34.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T07:09:34.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T07:09:34.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T07:09:34.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:09:34.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T07:09:34.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:09:34.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:09:34.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:09:34.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:09:34.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:09:34.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:09:34.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:09:34.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:09:34.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:09:34.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:09:34.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:09:34.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:09:34.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:09:34.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T07:09:34.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T07:09:34.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T07:09:34.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T07:09:34.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T07:09:34.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T07:09:34.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T07:09:34.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T07:09:35.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T07:09:35.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T07:09:35.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-03T07:09:35.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:09:35.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-03T07:09:35.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:09:35.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:09:35.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:09:35.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:09:35.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:09:35.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:09:35.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:09:35.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:09:36.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:09:36.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:09:36.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:09:36.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:09:36.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:09:36.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:09:36.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:09:36.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-06-03T07:09:36.398 INFO:tasks.workunit.client.0.smithi039.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-06-03T07:09:36.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:09:36.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-06-03T07:09:36.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-06-03T07:09:36.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-03T07:09:36.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:09:36.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:09:36.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-06-03T07:09:36.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-06-03T07:09:36.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-03T07:09:36.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:09:36.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:09:36.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-06-03T07:09:36.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-06-03T07:09:36.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-03T07:09:36.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:09:36.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:09:36.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-06-03T07:09:36.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-06-03T07:09:36.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-03T07:09:36.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:09:36.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:09:36.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-06-03T07:09:36.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-06-03T07:09:36.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-03T07:09:36.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:09:36.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-06-03T07:09:36.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-06-03T07:09:36.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:09:36.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:09:36.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:09:36.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:09:36.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-06-03T07:09:36.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:09:36.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:09:36.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:09:36.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:09:36.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:09:36.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:09:36.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:09:36.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:09:36.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:09:36.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:09:36.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:09:36.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-06-03T07:09:36.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:09:36.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:09:36.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:09:36.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:09:36.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:09:36.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-06-03T07:09:38.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:09:38.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:09:38.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:09:38.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:09:38.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:09:38.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:09:38.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:09:38.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:09:38.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:09:38.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:09:38.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:09:38.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:09:38.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:09:38.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:09:38.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:09:38.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:09:38.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:09:38.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:09:38.387 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:09:38.387 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:09:38.387 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:09:38.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:09:38.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:09:38.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:09:38.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:09:38.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:09:38.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:09:38.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:09:38.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:09:38.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:09:38.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:09:38.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:09:38.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:09:38.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:09:38.389 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:09:38.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:09:38.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:09:38.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:09:38.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:09:38.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:09:38.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:09:38.418 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:38.418+0000 7f9ba6015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:38.420 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:38.420+0000 7f9ba6015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:38.421 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:38.422+0000 7f9ba6015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:38.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:09:38.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:09:38.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:09:38.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:09:38.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:09:38.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:38.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:09:38.697 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:09:38.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:38.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:09:38.936 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:38.936+0000 7f9ba6015640 -1 Falling back to public interface 2024-06-03T07:09:38.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:39.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:39.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:39.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:09:39.994 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:09:39.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:39.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:09:40.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:41.242 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:41.243+0000 7f9ba6015640 -1 osd.0 129 log_to_monitors true 2024-06-03T07:09:41.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:41.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:41.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:09:41.297 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:09:41.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:41.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:09:41.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:42.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:42.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:42.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:09:42.739 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:09:42.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:42.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:09:43.040 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 133 up_thru 133 down_at 130 last_clean_interval [105,129) [v2:127.0.0.1:6802/985866297,v1:127.0.0.1:6803/985866297] [v2:127.0.0.1:6804/985866297,v1:127.0.0.1:6805/985866297] exists,up 3962cf66-f5f4-41c7-9b87-00f90eef0e73 2024-06-03T07:09:43.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:09:43.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:09:43.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:09:43.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:09:43.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:09:43.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:09:43.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:09:43.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:09:43.041 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:09:43.041 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:09:43.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:09:43.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:09:43.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:09:43.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:09:43.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:09:43.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:09:43.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:09:43.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:09:43.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:09:43.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:09:43.426 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:09:43.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:09:43.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:09:43.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:09:43.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650370 2024-06-03T07:09:43.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650370 2024-06-03T07:09:43.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650370' 2024-06-03T07:09:43.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:09:43.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:09:43.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846597 2024-06-03T07:09:43.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846597 2024-06-03T07:09:43.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650370 1-545460846597' 2024-06-03T07:09:43.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:09:43.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650370 2024-06-03T07:09:43.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:09:43.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:09:43.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650370 2024-06-03T07:09:43.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:09:43.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650370 2024-06-03T07:09:43.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650370' 2024-06-03T07:09:43.646 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 571230650370 2024-06-03T07:09:43.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:09:43.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 571230650370 2024-06-03T07:09:43.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:09:44.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:09:44.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:09:45.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 571230650370 2024-06-03T07:09:45.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:09:46.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T07:09:46.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:09:46.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650370 -lt 571230650370 2024-06-03T07:09:46.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:09:46.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-545460846597 2024-06-03T07:09:46.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:09:46.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:09:46.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-545460846597 2024-06-03T07:09:46.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:09:46.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846597 2024-06-03T07:09:46.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 545460846597' 2024-06-03T07:09:46.542 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 545460846597 2024-06-03T07:09:46.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:09:46.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846598 -lt 545460846597 2024-06-03T07:09:46.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:09:46.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:09:46.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:09:47.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:09:47.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:09:47.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:09:47.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:09:47.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:09:47.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:09:47.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:09:47.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:09:47.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:09:47.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:09:47.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:09:47.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:09:47.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:09:47.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:09:47.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:09:47.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-06-03T07:09:47.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:09:47.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:09:47.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:09:47.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:09:47.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-06-03T07:09:47.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:09:47.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:09:47.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:09:47.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:09:47.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:09:47.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:09:47.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:09:47.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:09:47.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:09:47.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:09:48.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:09:48.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-06-03T07:09:48.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:09:48.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:09:48.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:09:48.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:09:48.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:09:48.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-06-03T07:09:49.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:09:49.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:09:49.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:09:49.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:09:49.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:09:49.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:09:49.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:09:49.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:09:49.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:09:49.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:09:49.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:09:49.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:09:49.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:09:49.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:09:49.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:09:49.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:09:49.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:09:49.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:09:49.784 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:09:49.784 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:09:49.784 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:09:49.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:09:49.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:09:49.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:09:49.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:09:49.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:09:49.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:09:49.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:09:49.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:09:49.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:09:49.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:09:49.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:09:49.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:09:49.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:09:49.787 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:09:49.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:09:49.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:09:49.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:09:49.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:09:49.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:09:49.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:09:49.815 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:49.815+0000 7fe0c6ba7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:49.817 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:49.818+0000 7fe0c6ba7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:49.819 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:49.819+0000 7fe0c6ba7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:50.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:09:50.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:09:50.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:09:50.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:09:50.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:09:50.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:50.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:09:50.086 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:09:50.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:50.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:50.336 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:50.337+0000 7fe0c6ba7640 -1 Falling back to public interface 2024-06-03T07:09:50.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:51.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:51.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:51.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:09:51.377 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:09:51.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:51.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:51.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:52.662 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:52.662+0000 7fe0c6ba7640 -1 osd.1 134 log_to_monitors true 2024-06-03T07:09:52.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:52.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:52.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:09:52.683 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:09:52.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:52.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:53.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:09:54.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:09:54.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:09:54.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:09:54.042 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:09:54.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:09:54.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:09:54.341 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 138 up_thru 138 down_at 135 last_clean_interval [127,134) [v2:127.0.0.1:6810/1343064136,v1:127.0.0.1:6811/1343064136] [v2:127.0.0.1:6812/1343064136,v1:127.0.0.1:6813/1343064136] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:09:54.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:09:54.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:09:54.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:09:54.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:09:54.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:09:54.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:09:54.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:09:54.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:09:54.343 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:09:54.343 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:09:54.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:09:54.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:09:54.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:09:54.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:09:54.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:09:54.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:09:54.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:09:54.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:09:54.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:09:54.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:09:54.728 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:09:54.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:09:54.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:09:54.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:09:54.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650373 2024-06-03T07:09:54.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650373 2024-06-03T07:09:54.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650373' 2024-06-03T07:09:54.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:09:54.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:09:54.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=592705486850 2024-06-03T07:09:54.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 592705486850 2024-06-03T07:09:54.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650373 1-592705486850' 2024-06-03T07:09:54.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:09:54.944 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650373 2024-06-03T07:09:54.944 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:09:54.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:09:54.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650373 2024-06-03T07:09:54.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:09:54.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650373 2024-06-03T07:09:54.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650373' 2024-06-03T07:09:54.947 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 571230650373 2024-06-03T07:09:54.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:09:55.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650372 -lt 571230650373 2024-06-03T07:09:55.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:09:56.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:09:56.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:09:56.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650373 -lt 571230650373 2024-06-03T07:09:56.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:09:56.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-592705486850 2024-06-03T07:09:56.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:09:56.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:09:56.538 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-592705486850 2024-06-03T07:09:56.538 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:09:56.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=592705486850 2024-06-03T07:09:56.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 592705486850' 2024-06-03T07:09:56.539 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 592705486850 2024-06-03T07:09:56.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:09:56.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 592705486850 -lt 592705486850 2024-06-03T07:09:56.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:09:56.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:09:56.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:09:57.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:09:57.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:09:57.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:09:57.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:09:57.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:09:57.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:09:57.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:09:57.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:09:57.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:09:57.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:09:57.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:09:57.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:09:57.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:09:57.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:09:57.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:09:57.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-06-03T07:09:57.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:09:57.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-06-03T07:09:57.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-06-03T07:09:57.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-03T07:09:57.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:09:57.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-06-03T07:09:57.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:09:57.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:09:57.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:09:57.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:09:57.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-06-03T07:09:57.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:09:57.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:09:57.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:09:57.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:09:57.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:09:57.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:09:57.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:09:57.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:09:57.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:09:57.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:09:58.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:09:58.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-06-03T07:09:58.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:09:58.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:09:58.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:09:58.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:09:58.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:09:58.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-06-03T07:09:59.274 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-03T07:09:59.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:09:59.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:09:59.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:09:59.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:09:59.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:09:59.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:09:59.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:09:59.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:09:59.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:09:59.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:09:59.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:09:59.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:09:59.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:09:59.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:09:59.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:09:59.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:09:59.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:09:59.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:09:59.796 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:09:59.796 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:09:59.796 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:09:59.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:09:59.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:09:59.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:09:59.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:09:59.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:09:59.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:09:59.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:09:59.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:09:59.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:09:59.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:09:59.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:09:59.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:09:59.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:09:59.798 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:09:59.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:09:59.798 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:09:59.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:09:59.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:09:59.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:09:59.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:09:59.827 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:59.827+0000 7f5ad8614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:59.831 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:59.829+0000 7f5ad8614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:09:59.831 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:09:59.830+0000 7f5ad8614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:00.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:10:00.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:10:00.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:10:00.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:10:00.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:10:00.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:00.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:10:00.101 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:10:00.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:00.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:10:00.347 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:00.347+0000 7f5ad8614640 -1 Falling back to public interface 2024-06-03T07:10:00.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:10:01.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:10:01.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:01.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:10:01.414 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:10:01.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:01.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:10:01.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:10:02.643 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:02.643+0000 7f5ad8614640 -1 osd.1 140 log_to_monitors true 2024-06-03T07:10:02.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:10:02.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:02.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:10:02.715 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:10:02.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:02.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:10:03.141 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 144 up_thru 138 down_at 141 last_clean_interval [138,140) [v2:127.0.0.1:6810/2689054825,v1:127.0.0.1:6811/2689054825] [v2:127.0.0.1:6812/2689054825,v1:127.0.0.1:6813/2689054825] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:10:03.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:10:03.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:10:03.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:10:03.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:10:03.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:10:03.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:10:03.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:10:03.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:10:03.143 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:10:03.143 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:10:03.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:10:03.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:10:03.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:10:03.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:10:03.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:10:03.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:10:03.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:10:03.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:10:03.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:10:03.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:10:03.695 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:10:03.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:10:03.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:10:03.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:10:03.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650376 2024-06-03T07:10:03.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650376 2024-06-03T07:10:03.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650376' 2024-06-03T07:10:03.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:10:03.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:10:03.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=618475290626 2024-06-03T07:10:03.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 618475290626 2024-06-03T07:10:03.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650376 1-618475290626' 2024-06-03T07:10:03.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:10:03.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650376 2024-06-03T07:10:03.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:10:03.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:10:03.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650376 2024-06-03T07:10:03.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:10:03.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650376 2024-06-03T07:10:03.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650376' 2024-06-03T07:10:03.993 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 571230650376 2024-06-03T07:10:03.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:10:04.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650375 -lt 571230650376 2024-06-03T07:10:04.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:10:05.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:10:05.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:10:05.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650376 -lt 571230650376 2024-06-03T07:10:05.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:10:05.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-618475290626 2024-06-03T07:10:05.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:10:05.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:10:05.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-618475290626 2024-06-03T07:10:05.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:10:05.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=618475290626 2024-06-03T07:10:05.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 618475290626' 2024-06-03T07:10:05.604 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 618475290626 2024-06-03T07:10:05.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:10:05.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290626 -lt 618475290626 2024-06-03T07:10:05.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:10:05.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:05.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:06.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:10:06.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:10:06.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:10:06.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:10:06.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:10:06.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:10:06.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:10:06.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:10:06.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:10:06.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:10:06.538 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:06.538 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:06.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:10:06.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:10:06.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:10:06.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:10:06.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-06-03T07:10:06.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-06-03T07:10:06.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-03T07:10:06.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:10:06.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-06-03T07:10:06.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:10:06.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:10:06.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:10:06.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:10:06.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-06-03T07:10:06.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:10:06.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:10:06.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:10:06.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:10:06.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:10:06.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:10:06.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:10:06.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:10:06.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:10:06.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:10:07.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:10:07.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-06-03T07:10:07.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:10:07.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:10:07.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:10:07.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:10:07.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:10:07.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-06-03T07:10:08.474 INFO:tasks.workunit.client.0.smithi039.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-06-03T07:10:09.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:10:09.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:10:09.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:10:09.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:10:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:10:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:10:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:10:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:10:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:10:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:10:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:10:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:10:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:10:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:10:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:10:09.001 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:10:09.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:10:09.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:10:09.002 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:10:09.002 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:10:09.002 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:10:09.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:10:09.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:10:09.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:10:09.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:10:09.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:10:09.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:10:09.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:10:09.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:10:09.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:10:09.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:10:09.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:10:09.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:10:09.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:10:09.004 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:10:09.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:10:09.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:10:09.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:10:09.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:10:09.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:10:09.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:10:09.032 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:09.032+0000 7ff8cac1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:09.034 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:09.035+0000 7ff8cac1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:09.036 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:09.036+0000 7ff8cac1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:09.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:10:09.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:10:09.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:10:09.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:10:09.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:10:09.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:09.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:10:09.303 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:10:09.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:09.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:10:09.551 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:09.552+0000 7ff8cac1d640 -1 Falling back to public interface 2024-06-03T07:10:09.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:10:10.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:10:10.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:10.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:10:10.616 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:10:10.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:10.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:10:10.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:10:11.814 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:11.815+0000 7ff8cac1d640 -1 osd.0 145 log_to_monitors true 2024-06-03T07:10:11.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:10:11.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:11.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:10:11.920 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:10:11.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:11.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:10:12.218 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 149 up_thru 144 down_at 146 last_clean_interval [133,145) [v2:127.0.0.1:6802/2223608845,v1:127.0.0.1:6803/2223608845] [v2:127.0.0.1:6804/2223608845,v1:127.0.0.1:6805/2223608845] exists,up 3962cf66-f5f4-41c7-9b87-00f90eef0e73 2024-06-03T07:10:12.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:10:12.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:10:12.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:10:12.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:10:12.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:10:12.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:10:12.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:10:12.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:10:12.219 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:10:12.219 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:10:12.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:10:12.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:10:12.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:10:12.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:10:12.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:10:12.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:10:12.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:10:12.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:10:12.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:10:12.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:10:12.627 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:10:12.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:10:12.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:10:12.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:10:12.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=639950127106 2024-06-03T07:10:12.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 639950127106 2024-06-03T07:10:12.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127106' 2024-06-03T07:10:12.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:10:12.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:10:12.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=618475290629 2024-06-03T07:10:12.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 618475290629 2024-06-03T07:10:12.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127106 1-618475290629' 2024-06-03T07:10:12.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:10:12.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-639950127106 2024-06-03T07:10:12.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:10:12.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:10:12.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-639950127106 2024-06-03T07:10:12.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:10:12.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=639950127106 2024-06-03T07:10:12.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2024-06-03T07:10:12.926 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 639950127106 2024-06-03T07:10:12.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:10:13.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650377 -lt 639950127106 2024-06-03T07:10:13.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:10:14.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:10:14.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:10:14.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127106 -lt 639950127106 2024-06-03T07:10:14.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:10:14.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-618475290629 2024-06-03T07:10:14.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:10:14.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:10:14.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-618475290629 2024-06-03T07:10:14.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:10:14.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=618475290629 2024-06-03T07:10:14.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 618475290629' 2024-06-03T07:10:14.520 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 618475290629 2024-06-03T07:10:14.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:10:14.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290629 -lt 618475290629 2024-06-03T07:10:14.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:10:14.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:14.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:15.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:10:15.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:10:15.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:10:15.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:10:15.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:10:15.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:10:15.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:10:15.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:10:15.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:10:15.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:10:15.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:15.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:15.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:10:15.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-03T07:10:15.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T07:10:15.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-03T07:10:15.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:10:15.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T07:10:15.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:10:15.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:10:15.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:10:15.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:10:15.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:10:15.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:10:15.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:10:15.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:10:16.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:10:16.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:10:16.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:16.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:16.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:10:16.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T07:10:16.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T07:10:16.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T07:10:16.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T07:10:16.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T07:10:16.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T07:10:16.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T07:10:16.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T07:10:16.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T07:10:16.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T07:10:16.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T07:10:16.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:10:16.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T07:10:17.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:10:17.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:10:17.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:10:17.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:10:17.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:10:17.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:10:17.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:10:17.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:10:17.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:10:17.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:10:17.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:17.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:17.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:10:17.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:10:17.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:10:17.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-06-03T07:10:17.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-06-03T07:10:17.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:10:17.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:10:17.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:10:17.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:10:17.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-06-03T07:10:17.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:10:17.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:10:17.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:10:17.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:10:17.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:10:17.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:10:17.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:10:17.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:10:17.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:10:17.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:10:18.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:10:18.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-06-03T07:10:18.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:10:18.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:10:18.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:10:18.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:10:18.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:10:18.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-06-03T07:10:19.217 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-03T07:10:19.218 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-03T07:10:19.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:10:19.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:10:19.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:10:19.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:10:19.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:10:19.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:10:19.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:10:19.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:10:19.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:10:19.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:10:19.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:10:19.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:10:19.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:10:19.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:10:19.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:10:19.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:10:19.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:10:19.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:10:19.737 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:10:19.737 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:10:19.737 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:10:19.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:10:19.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:10:19.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:10:19.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:10:19.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:10:19.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:10:19.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:10:19.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:10:19.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:10:19.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:10:19.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:10:19.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:10:19.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:10:19.738 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:10:19.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:10:19.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:10:19.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:10:19.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:10:19.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:10:19.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:10:19.767 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:19.768+0000 7fdd44f6c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:19.770 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:19.770+0000 7fdd44f6c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:19.771 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:19.772+0000 7fdd44f6c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:20.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:10:20.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:10:20.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:10:20.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:10:20.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:10:20.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:20.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:10:20.037 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:10:20.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:20.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:10:20.287 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:20.288+0000 7fdd44f6c640 -1 Falling back to public interface 2024-06-03T07:10:20.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:10:21.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:10:21.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:21.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:10:21.333 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:10:21.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:21.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:10:21.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:10:22.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:10:22.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:22.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:10:22.633 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:10:22.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:22.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:10:22.741 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:22.741+0000 7fdd44f6c640 -1 osd.1 150 log_to_monitors true 2024-06-03T07:10:22.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:10:23.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:10:23.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:23.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:10:23.953 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:10:23.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:23.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:10:24.340 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [144,150) [v2:127.0.0.1:6810/133875879,v1:127.0.0.1:6811/133875879] [v2:127.0.0.1:6812/133875879,v1:127.0.0.1:6813/133875879] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:10:24.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:10:24.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:10:24.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:10:24.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:10:24.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:10:24.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:10:24.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:10:24.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:10:24.341 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:10:24.341 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:10:24.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:10:24.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:10:24.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:10:24.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:10:24.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:10:24.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:10:24.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:10:24.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:10:24.438 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:10:24.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:10:24.725 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:10:24.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:10:24.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:10:24.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:10:24.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=639950127109 2024-06-03T07:10:24.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 639950127109 2024-06-03T07:10:24.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127109' 2024-06-03T07:10:24.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:10:24.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:10:24.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=661424963586 2024-06-03T07:10:24.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 661424963586 2024-06-03T07:10:24.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127109 1-661424963586' 2024-06-03T07:10:24.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:10:24.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-639950127109 2024-06-03T07:10:24.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:10:24.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:10:24.943 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-639950127109 2024-06-03T07:10:24.943 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:10:24.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=639950127109 2024-06-03T07:10:24.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 639950127109' 2024-06-03T07:10:24.944 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 639950127109 2024-06-03T07:10:24.944 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:10:25.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127108 -lt 639950127109 2024-06-03T07:10:25.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:10:26.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:10:26.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:10:26.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127109 -lt 639950127109 2024-06-03T07:10:26.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:10:26.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-661424963586 2024-06-03T07:10:26.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:10:26.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:10:26.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-661424963586 2024-06-03T07:10:26.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:10:26.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=661424963586 2024-06-03T07:10:26.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2024-06-03T07:10:26.537 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 661424963586 2024-06-03T07:10:26.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:10:26.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 661424963586 -lt 661424963586 2024-06-03T07:10:26.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:10:26.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:26.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:27.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:10:27.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:10:27.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:10:27.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:10:27.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:10:27.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:10:27.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:10:27.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:10:27.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:10:27.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:10:27.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:27.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:27.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:10:27.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:10:27.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:10:27.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:10:27.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-06-03T07:10:27.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-06-03T07:10:27.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-03T07:10:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:10:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-06-03T07:10:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-06-03T07:10:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-03T07:10:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:10:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:10:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:10:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:10:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-03T07:10:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:10:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:10:27.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:10:27.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:10:27.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:10:27.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:10:27.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:10:27.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:10:27.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:10:27.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:10:28.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:10:28.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-03T07:10:28.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:10:28.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:10:28.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:10:28.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:10:28.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:10:28.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-03T07:10:29.277 INFO:tasks.workunit.client.0.smithi039.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-06-03T07:10:29.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:10:29.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:10:29.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:10:29.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:10:29.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:10:29.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:10:29.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:10:29.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:10:29.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:10:29.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:10:29.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:10:29.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:10:29.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:10:29.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:10:29.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:10:29.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:10:29.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:10:29.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:10:29.902 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:10:29.902 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:10:29.902 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:10:29.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:10:29.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:10:29.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:10:29.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:10:29.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:10:29.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:10:29.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:10:29.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:10:29.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:10:29.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:10:29.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:10:29.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:10:29.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:10:29.905 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:10:29.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:10:29.905 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:10:29.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:10:29.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:10:29.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:10:29.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:10:29.934 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:29.934+0000 7fa212ba7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:29.938 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:29.936+0000 7fa212ba7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:29.938 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:29.938+0000 7fa212ba7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:30.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:10:30.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:10:30.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:10:30.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:10:30.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:10:30.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:30.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:10:30.217 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:10:30.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:30.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:10:30.452 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:30.452+0000 7fa212ba7640 -1 Falling back to public interface 2024-06-03T07:10:30.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:10:31.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:10:31.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:31.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:10:31.508 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:10:31.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:31.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:10:31.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:10:32.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:10:32.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:32.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:10:32.812 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:10:32.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:32.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:10:32.835 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:32.836+0000 7fa212ba7640 -1 osd.0 156 log_to_monitors true 2024-06-03T07:10:33.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:10:34.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:10:34.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:34.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:10:34.234 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:10:34.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:34.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:10:34.578 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 160 up_thru 160 down_at 157 last_clean_interval [149,156) [v2:127.0.0.1:6802/2474418234,v1:127.0.0.1:6803/2474418234] [v2:127.0.0.1:6804/2474418234,v1:127.0.0.1:6805/2474418234] exists,up 3962cf66-f5f4-41c7-9b87-00f90eef0e73 2024-06-03T07:10:34.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:10:34.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:10:34.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:10:34.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:10:34.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:10:34.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:10:34.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:10:34.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:10:34.579 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:10:34.579 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:10:34.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:10:34.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:10:34.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:10:34.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:10:34.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:10:34.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:10:34.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:10:34.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:10:34.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:10:34.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:10:34.967 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:10:34.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:10:34.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:10:34.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:10:35.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=687194767362 2024-06-03T07:10:35.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 687194767362 2024-06-03T07:10:35.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767362' 2024-06-03T07:10:35.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:10:35.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:10:35.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=661424963589 2024-06-03T07:10:35.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 661424963589 2024-06-03T07:10:35.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767362 1-661424963589' 2024-06-03T07:10:35.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:10:35.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-687194767362 2024-06-03T07:10:35.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:10:35.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:10:35.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-687194767362 2024-06-03T07:10:35.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:10:35.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=687194767362 2024-06-03T07:10:35.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 687194767362' 2024-06-03T07:10:35.188 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 687194767362 2024-06-03T07:10:35.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:10:35.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767362 -lt 687194767362 2024-06-03T07:10:35.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:10:35.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-661424963589 2024-06-03T07:10:35.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:10:35.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:10:35.507 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-661424963589 2024-06-03T07:10:35.507 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:10:35.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=661424963589 2024-06-03T07:10:35.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 661424963589' 2024-06-03T07:10:35.509 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 661424963589 2024-06-03T07:10:35.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:10:35.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 661424963589 -lt 661424963589 2024-06-03T07:10:35.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:10:35.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:35.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:36.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:10:36.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:10:36.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:10:36.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:10:36.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:10:36.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:10:36.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:10:36.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:10:36.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:10:36.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:10:36.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:36.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:36.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:10:36.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:10:36.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:10:36.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-03T07:10:36.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:10:36.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:10:36.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:10:36.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:10:36.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-03T07:10:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:10:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:10:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:10:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:10:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:10:36.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:10:36.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:10:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:10:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:10:36.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:10:37.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:10:37.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-03T07:10:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:10:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:10:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:10:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:10:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:10:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-03T07:10:37.992 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-03T07:10:37.993 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-03T07:10:38.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:10:38.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:10:38.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:10:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:10:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:10:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:10:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:10:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:10:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:10:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:10:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:10:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:10:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:10:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:10:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:10:38.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:10:38.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:10:38.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:10:38.534 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:10:38.534 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:10:38.534 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:10:38.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:10:38.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:10:38.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:10:38.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:10:38.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:10:38.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:10:38.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:10:38.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:10:38.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:10:38.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:10:38.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:10:38.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:10:38.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:10:38.536 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:10:38.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:10:38.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:10:38.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:10:38.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:10:38.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:10:38.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:10:38.564 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:38.564+0000 7fa99b82c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:38.566 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:38.567+0000 7fa99b82c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:38.568 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:38.568+0000 7fa99b82c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:38.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:10:38.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:10:38.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:10:38.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:10:38.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:10:38.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:38.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:10:38.840 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:10:38.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:38.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:10:39.083 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:39.083+0000 7fa99b82c640 -1 Falling back to public interface 2024-06-03T07:10:39.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:10:40.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:10:40.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:40.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:10:40.133 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:10:40.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:40.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:10:40.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:10:41.197 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:41.197+0000 7fa99b82c640 -1 osd.1 161 log_to_monitors true 2024-06-03T07:10:41.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:10:41.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:41.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:10:41.451 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:10:41.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:41.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:10:41.817 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 165 up_thru 165 down_at 162 last_clean_interval [154,161) [v2:127.0.0.1:6810/1669898339,v1:127.0.0.1:6811/1669898339] [v2:127.0.0.1:6812/1669898339,v1:127.0.0.1:6813/1669898339] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:10:41.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:10:41.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:10:41.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:10:41.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:10:41.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:10:41.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:10:41.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:10:41.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:10:41.819 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:10:41.819 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:10:41.819 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:10:41.819 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:10:41.819 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:10:41.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:10:41.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:10:41.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:10:41.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:10:41.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:10:41.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:10:42.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:10:42.229 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:10:42.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:10:42.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:10:42.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:10:42.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=687194767364 2024-06-03T07:10:42.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 687194767364 2024-06-03T07:10:42.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767364' 2024-06-03T07:10:42.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:10:42.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:10:42.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=708669603842 2024-06-03T07:10:42.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 708669603842 2024-06-03T07:10:42.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767364 1-708669603842' 2024-06-03T07:10:42.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:10:42.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-687194767364 2024-06-03T07:10:42.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:10:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:10:42.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-687194767364 2024-06-03T07:10:42.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:10:42.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=687194767364 2024-06-03T07:10:42.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 687194767364' 2024-06-03T07:10:42.551 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 687194767364 2024-06-03T07:10:42.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:10:42.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767363 -lt 687194767364 2024-06-03T07:10:42.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:10:43.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:10:43.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:10:44.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767365 -lt 687194767364 2024-06-03T07:10:44.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:10:44.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-708669603842 2024-06-03T07:10:44.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:10:44.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:10:44.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-708669603842 2024-06-03T07:10:44.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:10:44.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=708669603842 2024-06-03T07:10:44.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 708669603842' 2024-06-03T07:10:44.142 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 708669603842 2024-06-03T07:10:44.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:10:44.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 708669603842 -lt 708669603842 2024-06-03T07:10:44.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:10:44.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:44.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:44.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:10:44.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:10:44.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:10:44.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:10:44.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:10:44.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:10:44.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:10:44.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:10:45.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:10:45.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:10:45.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:45.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:45.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:10:45.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:10:45.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:10:45.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:10:45.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-06-03T07:10:45.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-06-03T07:10:45.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-03T07:10:45.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:10:45.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-06-03T07:10:45.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-06-03T07:10:45.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-03T07:10:45.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:10:45.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:10:45.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:10:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:10:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-03T07:10:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:10:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:10:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:10:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:10:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:10:45.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:10:45.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:10:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:10:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:10:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:10:45.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:10:45.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-03T07:10:45.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:10:45.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:10:45.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:10:45.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:10:45.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:10:45.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-03T07:10:46.870 INFO:tasks.workunit.client.0.smithi039.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-06-03T07:10:47.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:10:47.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:10:47.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:10:47.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:10:47.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:10:47.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:10:47.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:10:47.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:10:47.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:10:47.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:10:47.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:10:47.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:10:47.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:10:47.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:10:47.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:10:47.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:10:47.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:10:47.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:10:47.465 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:10:47.465 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:10:47.465 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:10:47.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:10:47.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:10:47.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:10:47.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:10:47.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:10:47.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:10:47.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:10:47.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:10:47.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:10:47.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:10:47.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:10:47.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:10:47.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:10:47.468 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:10:47.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:10:47.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:10:47.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:10:47.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:10:47.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:10:47.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:10:47.496 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:47.496+0000 7fa59f612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:47.509 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:47.509+0000 7fa59f612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:47.510 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:47.511+0000 7fa59f612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:47.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:10:47.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:10:47.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:10:47.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:10:47.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:10:47.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:47.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:10:47.776 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:10:47.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:47.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:10:48.024 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:48.025+0000 7fa59f612640 -1 Falling back to public interface 2024-06-03T07:10:48.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:10:49.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:10:49.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:49.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:10:49.110 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:10:49.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:49.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:10:49.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:10:50.345 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:50.345+0000 7fa59f612640 -1 osd.0 167 log_to_monitors true 2024-06-03T07:10:50.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:10:50.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:10:50.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:10:50.413 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:10:50.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:10:50.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:10:50.831 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 171 up_thru 166 down_at 168 last_clean_interval [160,167) [v2:127.0.0.1:6802/2805016602,v1:127.0.0.1:6803/2805016602] [v2:127.0.0.1:6804/2805016602,v1:127.0.0.1:6805/2805016602] exists,up 3962cf66-f5f4-41c7-9b87-00f90eef0e73 2024-06-03T07:10:50.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:10:50.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:10:50.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:10:50.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:10:50.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:10:50.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:10:50.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:10:50.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:10:50.832 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:10:50.832 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:10:50.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:10:50.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:10:50.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:10:50.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:10:50.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:10:50.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:10:50.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:10:50.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:10:50.929 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:10:51.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:10:51.309 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:10:51.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:10:51.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:10:51.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:10:51.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407618 2024-06-03T07:10:51.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407618 2024-06-03T07:10:51.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407618' 2024-06-03T07:10:51.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:10:51.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:10:51.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=708669603845 2024-06-03T07:10:51.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 708669603845 2024-06-03T07:10:51.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407618 1-708669603845' 2024-06-03T07:10:51.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:10:51.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407618 2024-06-03T07:10:51.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:10:51.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:10:51.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407618 2024-06-03T07:10:51.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:10:51.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407618 2024-06-03T07:10:51.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407618' 2024-06-03T07:10:51.630 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 734439407618 2024-06-03T07:10:51.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:10:52.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 734439407618 2024-06-03T07:10:52.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:10:53.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:10:53.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:10:53.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 734439407618 2024-06-03T07:10:53.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:10:54.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T07:10:54.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:10:54.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407618 -lt 734439407618 2024-06-03T07:10:54.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:10:54.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-708669603845 2024-06-03T07:10:54.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:10:54.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:10:54.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-708669603845 2024-06-03T07:10:54.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:10:54.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=708669603845 2024-06-03T07:10:54.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 708669603845' 2024-06-03T07:10:54.678 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 708669603845 2024-06-03T07:10:54.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:10:54.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 708669603845 -lt 708669603845 2024-06-03T07:10:54.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:10:54.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:54.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:55.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:10:55.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:10:55.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:10:55.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:10:55.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:10:55.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:10:55.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:10:55.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:10:55.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:10:55.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:10:55.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:55.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:56.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:10:56.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-03T07:10:56.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T07:10:56.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-03T07:10:56.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:10:56.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T07:10:56.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:10:56.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:10:56.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:10:56.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:10:56.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:10:56.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:10:56.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:10:56.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:10:56.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:10:56.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:10:56.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:56.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:56.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:10:56.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T07:10:56.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T07:10:56.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T07:10:56.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T07:10:56.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T07:10:56.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T07:10:56.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T07:10:56.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T07:10:57.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T07:10:57.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T07:10:57.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T07:10:57.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:10:57.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T07:10:57.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:10:57.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:10:57.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:10:57.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:10:57.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:10:57.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:10:57.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:10:57.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:10:57.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:10:57.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:10:57.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:10:57.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:10:57.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:10:57.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:10:57.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:10:57.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-03T07:10:57.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:10:57.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:10:57.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:10:57.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:10:57.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-03T07:10:57.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:10:57.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:10:57.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:10:57.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:10:57.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:10:57.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:10:57.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:10:57.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:10:57.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:10:57.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:10:58.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:10:58.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-03T07:10:58.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:10:58.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:10:58.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:10:58.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:10:58.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:10:58.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-03T07:10:59.166 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-03T07:10:59.166 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-03T07:10:59.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:10:59.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:10:59.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:10:59.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:10:59.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:10:59.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:10:59.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:10:59.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:10:59.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:10:59.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:10:59.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:10:59.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:10:59.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:10:59.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:10:59.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:10:59.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:10:59.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:10:59.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:10:59.708 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:10:59.708 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:10:59.708 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:10:59.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:10:59.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:10:59.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:10:59.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:10:59.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:10:59.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:10:59.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:10:59.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:10:59.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:10:59.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:10:59.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:10:59.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:10:59.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:10:59.710 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:10:59.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:10:59.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:10:59.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:10:59.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:10:59.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:10:59.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:10:59.738 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:59.738+0000 7f707b60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:59.740 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:59.741+0000 7f707b60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:10:59.742 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:10:59.742+0000 7f707b60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:00.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:11:00.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:11:00.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:11:00.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:11:00.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:11:00.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:00.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:11:00.011 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:11:00.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:00.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:11:00.256 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:00.257+0000 7f707b60f640 -1 Falling back to public interface 2024-06-03T07:11:00.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:11:01.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:11:01.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:01.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:11:01.316 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:11:01.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:01.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:11:01.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:11:02.466 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:02.466+0000 7f707b60f640 -1 osd.1 173 log_to_monitors true 2024-06-03T07:11:02.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:11:02.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:02.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:11:02.617 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:11:02.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:02.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:11:03.059 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 177 up_thru 171 down_at 174 last_clean_interval [165,173) [v2:127.0.0.1:6810/1662031827,v1:127.0.0.1:6811/1662031827] [v2:127.0.0.1:6812/1662031827,v1:127.0.0.1:6813/1662031827] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:11:03.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:11:03.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:11:03.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:11:03.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:11:03.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:11:03.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:11:03.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:11:03.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:11:03.060 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:11:03.061 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:11:03.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:11:03.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:11:03.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:11:03.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:11:03.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:11:03.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:11:03.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:11:03.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:11:03.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:11:03.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:11:03.611 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:11:03.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:11:03.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:11:03.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:11:03.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407621 2024-06-03T07:11:03.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407621 2024-06-03T07:11:03.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407621' 2024-06-03T07:11:03.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:11:03.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:11:03.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=760209211394 2024-06-03T07:11:03.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 760209211394 2024-06-03T07:11:03.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407621 1-760209211394' 2024-06-03T07:11:03.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:11:03.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407621 2024-06-03T07:11:03.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:11:03.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:11:03.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407621 2024-06-03T07:11:03.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:11:03.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407621 2024-06-03T07:11:03.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407621' 2024-06-03T07:11:03.851 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 734439407621 2024-06-03T07:11:03.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:11:04.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407620 -lt 734439407621 2024-06-03T07:11:04.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:11:05.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:11:05.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:11:05.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407621 -lt 734439407621 2024-06-03T07:11:05.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:11:05.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-760209211394 2024-06-03T07:11:05.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:11:05.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:11:05.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-760209211394 2024-06-03T07:11:05.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:11:05.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=760209211394 2024-06-03T07:11:05.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 760209211394' 2024-06-03T07:11:05.482 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 760209211394 2024-06-03T07:11:05.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:11:05.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 760209211394 -lt 760209211394 2024-06-03T07:11:05.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:11:05.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:11:05.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:11:06.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:11:06.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:11:06.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:11:06.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:11:06.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:11:06.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:11:06.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:11:06.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:11:06.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:11:06.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:11:06.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:11:06.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:11:06.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:11:06.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:11:06.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:11:06.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-06-03T07:11:06.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:11:06.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:11:06.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:11:06.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:11:06.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-06-03T07:11:06.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:11:06.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:11:06.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:11:06.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:11:06.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:11:06.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:11:06.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:11:06.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:11:06.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:11:06.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:11:07.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:11:07.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-06-03T07:11:07.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:11:07.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:11:07.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:11:07.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:11:07.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:11:07.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-06-03T07:11:07.971 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-03T07:11:07.972 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-03T07:11:08.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:11:08.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:11:08.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:11:08.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:11:08.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:11:08.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:11:08.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:11:08.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:11:08.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:11:08.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:11:08.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:11:08.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:11:08.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:11:08.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:11:08.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:11:08.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:11:08.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:11:08.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:11:08.490 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:11:08.490 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:11:08.490 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:11:08.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:11:08.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:11:08.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:11:08.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:11:08.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:11:08.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:11:08.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:11:08.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:11:08.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:11:08.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:11:08.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:11:08.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:11:08.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:11:08.492 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:11:08.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:11:08.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:11:08.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:11:08.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:11:08.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:11:08.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:11:08.519 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:08.519+0000 7fcb0861f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:08.521 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:08.521+0000 7fcb0861f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:08.522 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:08.523+0000 7fcb0861f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:08.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:11:08.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:11:08.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:11:08.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:11:08.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:11:08.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:08.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:11:08.791 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:11:08.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:08.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:11:09.035 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:09.036+0000 7fcb0861f640 -1 Falling back to public interface 2024-06-03T07:11:09.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:11:10.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:11:10.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:10.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:11:10.087 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:11:10.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:10.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:11:10.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:11:11.157 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:11.158+0000 7fcb0861f640 -1 osd.1 178 log_to_monitors true 2024-06-03T07:11:11.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:11:11.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:11.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:11:11.384 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:11:11.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:11.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:11:11.810 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 182 up_thru 177 down_at 179 last_clean_interval [177,178) [v2:127.0.0.1:6810/42539212,v1:127.0.0.1:6811/42539212] [v2:127.0.0.1:6812/42539212,v1:127.0.0.1:6813/42539212] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:11:11.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:11:11.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:11:11.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:11:11.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:11:11.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:11:11.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:11:11.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:11:11.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:11:11.811 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:11:11.811 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:11:11.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:11:11.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:11:11.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:11:11.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:11:11.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:11:11.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:11:11.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:11:11.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:11:11.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:11:12.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:11:12.371 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:11:12.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:11:12.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:11:12.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:11:12.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407624 2024-06-03T07:11:12.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407624 2024-06-03T07:11:12.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407624' 2024-06-03T07:11:12.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:11:12.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:11:12.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=781684047874 2024-06-03T07:11:12.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 781684047874 2024-06-03T07:11:12.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407624 1-781684047874' 2024-06-03T07:11:12.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:11:12.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407624 2024-06-03T07:11:12.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:11:12.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:11:12.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407624 2024-06-03T07:11:12.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:11:12.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407624 2024-06-03T07:11:12.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407624' 2024-06-03T07:11:12.710 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 734439407624 2024-06-03T07:11:12.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:11:13.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407623 -lt 734439407624 2024-06-03T07:11:13.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:11:14.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:11:14.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:11:14.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407624 -lt 734439407624 2024-06-03T07:11:14.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:11:14.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-781684047874 2024-06-03T07:11:14.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:11:14.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:11:14.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-781684047874 2024-06-03T07:11:14.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:11:14.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=781684047874 2024-06-03T07:11:14.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 781684047874' 2024-06-03T07:11:14.343 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 781684047874 2024-06-03T07:11:14.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:11:14.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 781684047874 -lt 781684047874 2024-06-03T07:11:14.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:11:14.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:11:14.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:11:15.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:11:15.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:11:15.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:11:15.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:11:15.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:11:15.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:11:15.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:11:15.035 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:11:15.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:11:15.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:11:15.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:11:15.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:11:15.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:11:15.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-03T07:11:15.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T07:11:15.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-03T07:11:15.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:11:15.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T07:11:15.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:11:15.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:11:15.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:11:15.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:11:15.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:11:15.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:11:15.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:11:15.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:11:16.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:11:16.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:11:16.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:11:16.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:11:16.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:11:16.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T07:11:16.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T07:11:16.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T07:11:16.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T07:11:16.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T07:11:16.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T07:11:16.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T07:11:16.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T07:11:16.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T07:11:16.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T07:11:16.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T07:11:16.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:11:16.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T07:11:17.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:11:17.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:11:17.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:11:17.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:11:17.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:11:17.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:11:17.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:11:17.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:11:17.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:11:17.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:11:17.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:11:17.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:11:17.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:11:17.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:11:17.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:11:17.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-03T07:11:17.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-06-03T07:11:17.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-06-03T07:11:17.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-03T07:11:17.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-03T07:11:17.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-06-03T07:11:17.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-06-03T07:11:17.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-06-03T07:11:17.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-06-03T07:11:17.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T07:11:17.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-06-03T07:11:17.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-06-03T07:11:18.095 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-06-03T07:11:18.095 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-06-03T07:11:18.113 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-06-03T07:11:18.113 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-06-03T07:11:18.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-06-03T07:11:18.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-03T07:11:18.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-03T07:11:18.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-06-03T07:11:18.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-03T07:11:18.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-03T07:11:18.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-03T07:11:18.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-06-03T07:11:18.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-03T07:11:18.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-03T07:11:18.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-03T07:11:18.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-06-03T07:11:18.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-03T07:11:18.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-06-03T07:11:18.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-03T07:11:18.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-06-03T07:11:18.124 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-06-03T07:11:18.124 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T07:11:18.422 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-03T07:11:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-06-03T07:11:18.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-03T07:11:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-03T07:11:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-03T07:11:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-06-03T07:11:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-03T07:11:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-03T07:11:18.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-06-03T07:11:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-03T07:11:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-06-03T07:11:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-03T07:11:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-03T07:11:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-03T07:11:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-03T07:11:18.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-03T07:11:18.426 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-03T07:11:18.426 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-03T07:11:18.426 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-03T07:11:18.426 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:11:18.426 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:11:18.426 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:11:18.426 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.1.asok 2024-06-03T07:11:18.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69989/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-06-03T07:11:18.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-03T07:11:18.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-03T07:11:18.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-03T07:11:18.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-06-03T07:11:18.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-03T07:11:18.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-03T07:11:18.518 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:11:18.518 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:11:18.518 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:11:18.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.1.asok 2024-06-03T07:11:18.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-03T07:11:18.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69989/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-06-03T07:11:18.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-06-03T07:11:18.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-03T07:11:18.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-03T07:11:18.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-06-03T07:11:18.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-03T07:11:18.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-03T07:11:18.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-03T07:11:18.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-06-03T07:11:18.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-03T07:11:18.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-03T07:11:18.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-03T07:11:18.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-06-03T07:11:18.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-03T07:11:18.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-06-03T07:11:18.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-03T07:11:18.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-06-03T07:11:18.612 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-06-03T07:11:18.612 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T07:11:18.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-03T07:11:18.903 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-06-03T07:11:18.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-03T07:11:18.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-03T07:11:18.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-03T07:11:18.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-06-03T07:11:18.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-03T07:11:18.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-03T07:11:18.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-06-03T07:11:18.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-03T07:11:18.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-06-03T07:11:18.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-03T07:11:18.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-06-03T07:11:18.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-03T07:11:18.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-03T07:11:18.905 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-03T07:11:18.905 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-06-03T07:11:18.905 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T07:11:18.905 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T07:11:18.906 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:11:18.906 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:11:18.906 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:11:18.906 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.0.asok 2024-06-03T07:11:18.906 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69989/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-06-03T07:11:18.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-03T07:11:18.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-03T07:11:18.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-03T07:11:18.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-06-03T07:11:18.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T07:11:18.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T07:11:18.997 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:11:18.998 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:11:18.998 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:11:18.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.0.asok 2024-06-03T07:11:18.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-03T07:11:18.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69989/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-06-03T07:11:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-06-03T07:11:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-03T07:11:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-03T07:11:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-06-03T07:11:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-03T07:11:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-03T07:11:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-03T07:11:19.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-06-03T07:11:19.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-03T07:11:19.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-03T07:11:19.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-03T07:11:19.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-06-03T07:11:19.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-03T07:11:19.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-06-03T07:11:19.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-03T07:11:19.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-06-03T07:11:19.093 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-06-03T07:11:19.093 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T07:11:19.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-03T07:11:19.387 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-06-03T07:11:19.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-03T07:11:19.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-03T07:11:19.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-03T07:11:19.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-06-03T07:11:19.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-03T07:11:19.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-03T07:11:19.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-06-03T07:11:19.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-03T07:11:19.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-06-03T07:11:19.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-03T07:11:19.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-06-03T07:11:19.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-03T07:11:19.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-03T07:11:19.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-03T07:11:19.390 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-06-03T07:11:19.390 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T07:11:19.390 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T07:11:19.390 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:11:19.390 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:11:19.390 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:11:19.390 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.0.asok 2024-06-03T07:11:19.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69989/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-06-03T07:11:19.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-03T07:11:19.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-03T07:11:19.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-03T07:11:19.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-06-03T07:11:19.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T07:11:19.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T07:11:19.482 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:11:19.482 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:11:19.482 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:11:19.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.0.asok 2024-06-03T07:11:19.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-03T07:11:19.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69989/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-06-03T07:11:19.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-06-03T07:11:19.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-03T07:11:19.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-03T07:11:19.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-06-03T07:11:19.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-03T07:11:19.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-03T07:11:19.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-03T07:11:19.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-06-03T07:11:19.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-03T07:11:19.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-03T07:11:19.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-03T07:11:19.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-06-03T07:11:19.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-03T07:11:19.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-06-03T07:11:19.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-03T07:11:19.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-06-03T07:11:19.576 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-06-03T07:11:19.576 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T07:11:19.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-03T07:11:19.871 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-06-03T07:11:19.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-03T07:11:19.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-03T07:11:19.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-03T07:11:19.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-06-03T07:11:19.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-03T07:11:19.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-03T07:11:19.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-06-03T07:11:19.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-03T07:11:19.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-06-03T07:11:19.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-03T07:11:19.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-03T07:11:19.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-03T07:11:19.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-03T07:11:19.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-03T07:11:19.874 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-03T07:11:19.874 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-03T07:11:19.874 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-03T07:11:19.874 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:11:19.874 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:11:19.874 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:11:19.874 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.1.asok 2024-06-03T07:11:19.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69989/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-06-03T07:11:19.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-03T07:11:19.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-03T07:11:19.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-03T07:11:19.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-06-03T07:11:19.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-03T07:11:19.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-03T07:11:19.966 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:11:19.966 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:11:19.966 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:11:19.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.1.asok 2024-06-03T07:11:19.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-03T07:11:19.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69989/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-06-03T07:11:20.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-06-03T07:11:20.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-06-03T07:11:20.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-06-03T07:11:20.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:11:20.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:11:20.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:11:20.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:11:20.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-06-03T07:07:24.364147+0000 2024-06-03T07:11:20.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-06-03T07:11:20.616 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 3.0 on osd.1 to scrub 2024-06-03T07:11:20.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-06-03T07:07:24.364147+0000 2024-06-03T07:11:20.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-03T07:11:20.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T07:07:24.364147+0000 2024-06-03T07:11:20.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T07:11:20.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:11:20.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:11:20.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:11:20.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:11:20.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:11:20.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:11:20.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:11:20.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:07:24.364147+0000 '>' 2024-06-03T07:07:24.364147+0000 2024-06-03T07:11:20.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:11:21.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:11:21.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:11:21.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:11:21.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:11:21.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:11:21.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:11:21.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:11:22.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:07:24.364147+0000 '>' 2024-06-03T07:07:24.364147+0000 2024-06-03T07:11:22.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:11:23.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:11:23.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:11:23.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:11:23.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:11:23.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:11:23.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:11:23.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:11:23.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:11:21.317195+0000 '>' 2024-06-03T07:07:24.364147+0000 2024-06-03T07:11:23.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:11:23.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-06-03T07:11:23.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-06-03T07:11:23.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-06-03T07:11:23.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-06-03T07:11:23.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-06-03T07:11:23.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-06-03T07:11:23.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-06-03T07:11:23.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-06-03T07:11:23.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-06-03T07:11:23.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-06-03T07:11:23.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-06-03T07:11:23.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-06-03T07:11:23.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-06-03T07:11:23.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-06-03T07:11:23.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-06-03T07:11:23.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-06-03T07:11:23.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-06-03T07:11:23.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-06-03T07:11:23.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-06-03T07:11:23.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-06-03T07:11:23.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-06-03T07:11:23.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-06-03T07:11:23.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-06-03T07:11:23.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-06-03T07:11:23.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:11:23.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-06-03T07:11:23.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-06-03T07:11:23.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-06-03T07:11:23.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-06-03T07:11:23.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-03T07:11:23.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-06-03T07:11:23.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-06-03T07:11:23.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-06-03T07:11:23.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=182 2024-06-03T07:11:23.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-06-03T07:11:23.784 INFO:tasks.workunit.client.0.smithi039.stderr: . as $in 2024-06-03T07:11:23.784 INFO:tasks.workunit.client.0.smithi039.stderr: | if type == "object" then 2024-06-03T07:11:23.785 INFO:tasks.workunit.client.0.smithi039.stderr: reduce keys[] as $key 2024-06-03T07:11:23.785 INFO:tasks.workunit.client.0.smithi039.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-03T07:11:23.785 INFO:tasks.workunit.client.0.smithi039.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-03T07:11:23.785 INFO:tasks.workunit.client.0.smithi039.stderr: else f 2024-06-03T07:11:23.785 INFO:tasks.workunit.client.0.smithi039.stderr: end; 2024-06-03T07:11:23.785 INFO:tasks.workunit.client.0.smithi039.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-03T07:11:23.785 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-03T07:11:23.785 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-03T07:11:23.785 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-03T07:11:23.785 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-03T07:11:23.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-06-03T07:11:23.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-03T07:11:23.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-06-03T07:11:23.809 INFO:tasks.workunit.client.0.smithi039.stderr: . as $in 2024-06-03T07:11:23.809 INFO:tasks.workunit.client.0.smithi039.stderr: | if type == "object" then 2024-06-03T07:11:23.809 INFO:tasks.workunit.client.0.smithi039.stderr: reduce keys[] as $key 2024-06-03T07:11:23.809 INFO:tasks.workunit.client.0.smithi039.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-03T07:11:23.809 INFO:tasks.workunit.client.0.smithi039.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-03T07:11:23.809 INFO:tasks.workunit.client.0.smithi039.stderr: else f 2024-06-03T07:11:23.809 INFO:tasks.workunit.client.0.smithi039.stderr: end; 2024-06-03T07:11:23.809 INFO:tasks.workunit.client.0.smithi039.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-03T07:11:23.809 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-03T07:11:23.809 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-03T07:11:23.810 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-03T07:11:23.810 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-03T07:11:23.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-06-03T07:11:23.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-03T07:11:23.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-03T07:11:23.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-03T07:11:23.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-06-03T07:11:23.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-06-03T07:11:23.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-06-03T07:11:23.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-06-03T07:11:23.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-06-03T07:11:23.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-03T07:11:23.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:11:23.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:11:23.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:11:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:11:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-03T07:11:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:11:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:11:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:11:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:11:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:11:23.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:11:23.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:11:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:11:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:11:23.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:11:24.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:11:24.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-03T07:11:24.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:11:24.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:11:24.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:11:24.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:11:24.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:11:24.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-03T07:11:25.126 INFO:tasks.workunit.client.0.smithi039.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-06-03T07:11:25.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:11:25.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:11:25.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:11:25.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:11:25.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:11:25.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:11:25.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:11:25.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:11:25.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:11:25.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:11:25.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:11:25.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:11:25.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:11:25.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:11:25.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:11:25.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:11:25.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:11:25.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:11:25.650 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:11:25.650 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:11:25.650 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:11:25.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:11:25.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:11:25.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:11:25.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:11:25.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:11:25.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:11:25.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:11:25.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:11:25.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:11:25.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:11:25.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:11:25.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:11:25.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:11:25.652 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:11:25.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:11:25.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:11:25.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:11:25.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:11:25.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:11:25.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:11:25.681 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:25.681+0000 7fb8e0d68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:25.683 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:25.684+0000 7fb8e0d68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:25.684 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:25.685+0000 7fb8e0d68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:25.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:11:25.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:11:25.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:11:25.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:11:25.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:11:25.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:25.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:11:25.951 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:11:25.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:25.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:11:26.199 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:26.200+0000 7fb8e0d68640 -1 Falling back to public interface 2024-06-03T07:11:26.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:11:27.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:11:27.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:27.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:11:27.246 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:11:27.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:27.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:11:27.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:11:28.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:11:28.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:28.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:11:28.548 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:11:28.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:28.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:11:28.581 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:28.582+0000 7fb8e0d68640 -1 osd.0 184 log_to_monitors true 2024-06-03T07:11:28.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:11:29.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:11:29.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:29.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:11:29.855 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:11:29.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:29.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:11:30.206 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 188 up_thru 188 down_at 185 last_clean_interval [171,184) [v2:127.0.0.1:6802/2966617654,v1:127.0.0.1:6803/2966617654] [v2:127.0.0.1:6804/2966617654,v1:127.0.0.1:6805/2966617654] exists,up 3962cf66-f5f4-41c7-9b87-00f90eef0e73 2024-06-03T07:11:30.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:11:30.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:11:30.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:11:30.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:11:30.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:11:30.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:11:30.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:11:30.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:11:30.207 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:11:30.207 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:11:30.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:11:30.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:11:30.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:11:30.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:11:30.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:11:30.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:11:30.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:11:30.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:11:30.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:11:30.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:11:30.592 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:11:30.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:11:30.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:11:30.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:11:30.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=807453851650 2024-06-03T07:11:30.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 807453851650 2024-06-03T07:11:30.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-807453851650' 2024-06-03T07:11:30.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:11:30.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:11:30.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=781684047878 2024-06-03T07:11:30.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 781684047878 2024-06-03T07:11:30.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-807453851650 1-781684047878' 2024-06-03T07:11:30.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:11:30.810 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-807453851650 2024-06-03T07:11:30.810 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:11:30.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:11:30.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-807453851650 2024-06-03T07:11:30.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:11:30.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=807453851650 2024-06-03T07:11:30.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 807453851650' 2024-06-03T07:11:30.812 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 807453851650 2024-06-03T07:11:30.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:11:31.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 807453851650 2024-06-03T07:11:31.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:11:32.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:11:32.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:11:32.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 807453851650 -lt 807453851650 2024-06-03T07:11:32.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:11:32.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-781684047878 2024-06-03T07:11:32.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:11:32.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:11:32.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-781684047878 2024-06-03T07:11:32.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:11:32.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=781684047878 2024-06-03T07:11:32.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 781684047878' 2024-06-03T07:11:32.404 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 781684047878 2024-06-03T07:11:32.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:11:32.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 781684047878 -lt 781684047878 2024-06-03T07:11:32.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:11:32.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:11:32.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:11:33.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:11:33.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:11:33.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:11:33.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:11:33.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:11:33.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:11:33.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:11:33.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:11:33.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:11:33.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:11:33.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:11:33.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:11:33.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:11:33.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:11:33.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:11:33.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-06-03T07:11:33.707 INFO:tasks.workunit.client.0.smithi039.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-06-03T07:11:33.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-06-03T07:11:33.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-06-03T07:11:33.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:11:33.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:11:33.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:11:33.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:11:33.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-06-03T07:11:33.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:11:33.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:11:33.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:11:33.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:11:33.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:11:33.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:11:33.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:11:33.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:11:33.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:11:33.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:11:34.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:11:34.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-06-03T07:11:34.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:11:34.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:11:34.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:11:34.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:11:34.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:11:34.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-06-03T07:11:34.846 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-03T07:11:34.846 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-03T07:11:35.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:11:35.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:11:35.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:11:35.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:11:35.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:11:35.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:11:35.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:11:35.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:11:35.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:11:35.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:11:35.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:11:35.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:11:35.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:11:35.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:11:35.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:11:35.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:11:35.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:11:35.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:11:35.368 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:11:35.368 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:11:35.368 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:11:35.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:11:35.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:11:35.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:11:35.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:11:35.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:11:35.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:11:35.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:11:35.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:11:35.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:11:35.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:11:35.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:11:35.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:11:35.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:11:35.371 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:11:35.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:11:35.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:11:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:11:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:11:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:11:35.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:11:35.400 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:35.401+0000 7fad62b2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:35.403 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:35.403+0000 7fad62b2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:35.404 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:35.404+0000 7fad62b2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:35.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:11:35.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:11:35.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:11:35.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:11:35.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:11:35.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:35.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:11:35.670 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:11:35.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:35.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:11:35.923 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:35.924+0000 7fad62b2f640 -1 Falling back to public interface 2024-06-03T07:11:35.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:11:36.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:11:36.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:36.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:11:36.965 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:11:36.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:36.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:11:37.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:11:38.029 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:38.029+0000 7fad62b2f640 -1 osd.1 190 log_to_monitors true 2024-06-03T07:11:38.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:11:38.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:38.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:11:38.260 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:11:38.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:38.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:11:38.625 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 194 up_thru 189 down_at 191 last_clean_interval [182,190) [v2:127.0.0.1:6810/3740414133,v1:127.0.0.1:6811/3740414133] [v2:127.0.0.1:6812/3740414133,v1:127.0.0.1:6813/3740414133] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:11:38.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:11:38.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:11:38.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:11:38.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:11:38.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:11:38.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:11:38.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:11:38.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:11:38.627 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:11:38.627 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:11:38.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:11:38.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:11:38.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:11:38.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:11:38.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:11:38.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:11:38.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:11:38.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:11:38.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:11:39.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:11:39.052 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:11:39.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:11:39.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:11:39.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:11:39.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=807453851653 2024-06-03T07:11:39.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 807453851653 2024-06-03T07:11:39.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-807453851653' 2024-06-03T07:11:39.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:11:39.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:11:39.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=833223655426 2024-06-03T07:11:39.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 833223655426 2024-06-03T07:11:39.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-807453851653 1-833223655426' 2024-06-03T07:11:39.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:11:39.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-807453851653 2024-06-03T07:11:39.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:11:39.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:11:39.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-807453851653 2024-06-03T07:11:39.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:11:39.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=807453851653 2024-06-03T07:11:39.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 807453851653' 2024-06-03T07:11:39.350 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 807453851653 2024-06-03T07:11:39.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:11:39.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 807453851653 -lt 807453851653 2024-06-03T07:11:39.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:11:39.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-833223655426 2024-06-03T07:11:39.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:11:39.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:11:39.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-833223655426 2024-06-03T07:11:39.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:11:39.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=833223655426 2024-06-03T07:11:39.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 833223655426' 2024-06-03T07:11:39.777 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 833223655426 2024-06-03T07:11:39.777 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:11:40.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 833223655426 2024-06-03T07:11:40.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:11:41.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:11:41.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:11:41.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 833223655426 2024-06-03T07:11:41.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:11:42.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T07:11:42.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:11:42.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 833223655426 -lt 833223655426 2024-06-03T07:11:42.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:11:42.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:11:42.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:11:43.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:11:43.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:11:43.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:11:43.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:11:43.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:11:43.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:11:43.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:11:43.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:11:43.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:11:43.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:11:43.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:11:43.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:11:43.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:11:43.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-03T07:11:43.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T07:11:43.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-03T07:11:43.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:11:43.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T07:11:43.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:11:43.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:11:43.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:11:43.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:11:43.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:11:43.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:11:43.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:11:43.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:11:44.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:11:44.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:11:44.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:11:44.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:11:44.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:11:44.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T07:11:44.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T07:11:44.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T07:11:44.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T07:11:44.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T07:11:44.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T07:11:44.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T07:11:44.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T07:11:44.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T07:11:44.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T07:11:44.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T07:11:44.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:11:44.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T07:11:45.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:11:45.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:11:45.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:11:45.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:11:45.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:11:45.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:11:45.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:11:45.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:11:45.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-03T07:11:45.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:11:45.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:11:45.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:11:45.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-03T07:11:45.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-03T07:11:45.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T07:11:45.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T07:11:45.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T07:11:45.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T07:11:45.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T07:11:45.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T07:11:45.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T07:11:46.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T07:11:46.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T07:11:46.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-03T07:11:46.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:11:46.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-03T07:11:46.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:11:46.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:11:46.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:11:46.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:11:46.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:11:46.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:11:46.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:11:46.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:11:46.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:11:46.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:11:46.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:11:46.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:11:47.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:11:47.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:11:47.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:11:47.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-06-03T07:11:47.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-03T07:11:47.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:11:47.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:11:47.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:11:47.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:11:47.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-03T07:11:47.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:11:47.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:11:47.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:11:47.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:11:47.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:11:47.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:11:47.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:11:47.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:11:47.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:11:47.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:11:47.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:11:47.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-03T07:11:47.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:11:47.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:11:47.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:11:47.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:11:47.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:11:47.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-03T07:11:48.722 INFO:tasks.workunit.client.0.smithi039.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-06-03T07:11:49.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:11:49.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:11:49.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:11:49.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:11:49.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:11:49.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:11:49.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:11:49.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:11:49.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:11:49.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:11:49.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:11:49.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:11:49.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:11:49.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:11:49.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:11:49.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:11:49.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:11:49.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:11:49.241 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:11:49.241 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:11:49.241 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:11:49.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:11:49.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:11:49.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:11:49.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:11:49.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:11:49.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:11:49.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:11:49.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:11:49.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:11:49.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:11:49.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:11:49.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:11:49.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:11:49.243 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:11:49.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:11:49.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:11:49.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:11:49.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:11:49.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:11:49.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:11:49.272 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:49.272+0000 7f3d5a199640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:49.274 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:49.274+0000 7f3d5a199640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:49.276 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:49.275+0000 7f3d5a199640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:49.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:11:49.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:11:49.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:11:49.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:11:49.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:11:49.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:49.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:11:49.546 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:11:49.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:49.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:11:49.790 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:49.789+0000 7f3d5a199640 -1 Falling back to public interface 2024-06-03T07:11:49.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:11:50.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:11:50.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:50.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:11:50.877 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:11:50.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:50.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:11:51.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:11:52.152 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:52.153+0000 7f3d5a199640 -1 osd.0 196 log_to_monitors true 2024-06-03T07:11:52.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:11:52.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:52.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:11:52.172 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:11:52.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:52.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:11:52.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:11:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:11:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:11:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:11:53.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:53.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:11:53.780 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 200 up_thru 200 down_at 197 last_clean_interval [188,196) [v2:127.0.0.1:6802/2362304929,v1:127.0.0.1:6803/2362304929] [v2:127.0.0.1:6804/2362304929,v1:127.0.0.1:6805/2362304929] exists,up 3962cf66-f5f4-41c7-9b87-00f90eef0e73 2024-06-03T07:11:53.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:11:53.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:11:53.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:11:53.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:11:53.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:11:53.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:11:53.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:11:53.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:11:53.780 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:11:53.780 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:11:53.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:11:53.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:11:53.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:11:53.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:11:53.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:11:53.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:11:53.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:11:53.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:11:53.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:11:54.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:11:54.165 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:11:54.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:11:54.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:11:54.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:11:54.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=858993459202 2024-06-03T07:11:54.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 858993459202 2024-06-03T07:11:54.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-858993459202' 2024-06-03T07:11:54.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:11:54.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:11:54.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=833223655430 2024-06-03T07:11:54.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 833223655430 2024-06-03T07:11:54.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-858993459202 1-833223655430' 2024-06-03T07:11:54.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:11:54.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-858993459202 2024-06-03T07:11:54.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:11:54.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:11:54.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-858993459202 2024-06-03T07:11:54.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:11:54.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=858993459202 2024-06-03T07:11:54.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 858993459202' 2024-06-03T07:11:54.385 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 858993459202 2024-06-03T07:11:54.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:11:54.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 858993459202 2024-06-03T07:11:54.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:11:55.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:11:55.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:11:55.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 858993459202 -lt 858993459202 2024-06-03T07:11:55.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:11:55.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-833223655430 2024-06-03T07:11:55.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:11:55.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:11:55.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-833223655430 2024-06-03T07:11:55.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:11:55.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=833223655430 2024-06-03T07:11:55.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 833223655430' 2024-06-03T07:11:55.982 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 833223655430 2024-06-03T07:11:55.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:11:56.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 833223655430 -lt 833223655430 2024-06-03T07:11:56.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:11:56.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:11:56.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:11:56.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:11:56.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:11:56.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:11:56.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:11:56.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:11:56.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:11:56.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:11:56.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:11:56.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:11:56.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:11:56.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:11:56.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:11:57.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:11:57.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:11:57.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:11:57.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-03T07:11:57.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:11:57.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:11:57.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:11:57.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:11:57.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-03T07:11:57.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:11:57.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:11:57.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:11:57.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:11:57.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:11:57.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:11:57.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:11:57.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:11:57.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:11:57.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:11:57.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:11:57.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-03T07:11:57.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:11:57.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:11:57.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:11:57.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:11:57.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:11:57.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-03T07:11:58.452 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-03T07:11:58.453 INFO:tasks.workunit.client.0.smithi039.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-03T07:11:58.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:11:58.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:11:58.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:11:58.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:11:58.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:11:58.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:11:58.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:11:58.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:11:58.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:11:58.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:11:58.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:11:58.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:11:58.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:11:58.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:11:58.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:11:58.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:11:58.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:11:58.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:11:58.970 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:11:58.970 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:11:58.970 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:11:58.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:11:58.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:11:58.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:11:58.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:11:58.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:11:58.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:11:58.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:11:58.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:11:58.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:11:58.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:11:58.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:11:58.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:11:58.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:11:58.972 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:11:58.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:11:58.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:11:58.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:11:58.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:11:58.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:11:58.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:11:59.001 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:59.002+0000 7fd45cc18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:59.004 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:59.004+0000 7fd45cc18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:59.006 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:59.006+0000 7fd45cc18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:11:59.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:11:59.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:11:59.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:11:59.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:11:59.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:11:59.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:11:59.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:11:59.275 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:11:59.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:11:59.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:11:59.521 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:11:59.522+0000 7fd45cc18640 -1 Falling back to public interface 2024-06-03T07:11:59.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:00.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:00.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:00.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:12:00.573 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:12:00.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:00.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:12:00.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:01.710 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:01.710+0000 7fd45cc18640 -1 osd.1 202 log_to_monitors true 2024-06-03T07:12:01.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:01.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:01.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:12:01.873 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:12:01.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:01.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:12:02.359 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 206 up_thru 201 down_at 203 last_clean_interval [194,202) [v2:127.0.0.1:6810/3687619134,v1:127.0.0.1:6811/3687619134] [v2:127.0.0.1:6812/3687619134,v1:127.0.0.1:6813/3687619134] exists,up b4579ed5-c309-4787-9209-8f3757d4856d 2024-06-03T07:12:02.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:12:02.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:12:02.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:12:02.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:12:02.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:12:02.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:12:02.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:12:02.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:12:02.462 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:12:02.463 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:12:02.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:12:02.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:12:02.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:12:02.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:12:02.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:12:02.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:12:02.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:12:02.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:12:02.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:12:02.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:12:02.920 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:12:02.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:12:02.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:12:02.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:12:03.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=858993459205 2024-06-03T07:12:03.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 858993459205 2024-06-03T07:12:03.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-858993459205' 2024-06-03T07:12:03.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:12:03.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:12:03.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=884763262978 2024-06-03T07:12:03.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 884763262978 2024-06-03T07:12:03.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-858993459205 1-884763262978' 2024-06-03T07:12:03.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:12:03.164 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-858993459205 2024-06-03T07:12:03.164 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:12:03.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:12:03.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-858993459205 2024-06-03T07:12:03.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:12:03.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=858993459205 2024-06-03T07:12:03.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 858993459205' 2024-06-03T07:12:03.167 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 858993459205 2024-06-03T07:12:03.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:12:03.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 858993459205 -lt 858993459205 2024-06-03T07:12:03.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:12:03.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-884763262978 2024-06-03T07:12:03.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:12:03.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:12:03.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-884763262978 2024-06-03T07:12:03.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:12:03.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=884763262978 2024-06-03T07:12:03.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 884763262978' 2024-06-03T07:12:03.464 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 884763262978 2024-06-03T07:12:03.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:12:03.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 884763262978 -lt 884763262978 2024-06-03T07:12:03.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:12:03.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:12:03.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:12:04.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:12:04.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:12:04.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:12:04.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:12:04.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:12:04.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:12:04.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:12:04.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:12:04.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:12:04.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:12:04.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:12:04.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:12:04.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:12:04.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:12:04.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:12:04.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-06-03T07:12:04.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-06-03T07:12:04.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-03T07:12:04.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-03T07:12:04.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-06-03T07:12:04.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-03T07:12:04.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-03T07:12:04.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-03T07:12:04.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-06-03T07:12:04.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-03T07:12:04.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-03T07:12:04.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-03T07:12:04.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-06-03T07:12:04.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-03T07:12:04.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-06-03T07:12:04.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-03T07:12:04.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-06-03T07:12:04.847 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-06-03T07:12:04.847 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T07:12:05.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-03T07:12:05.180 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-06-03T07:12:05.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-03T07:12:05.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-03T07:12:05.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-03T07:12:05.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-06-03T07:12:05.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-03T07:12:05.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-03T07:12:05.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-06-03T07:12:05.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-03T07:12:05.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-06-03T07:12:05.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-03T07:12:05.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-03T07:12:05.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-03T07:12:05.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-03T07:12:05.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-03T07:12:05.183 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-03T07:12:05.183 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-03T07:12:05.183 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-03T07:12:05.183 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:12:05.183 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:05.183 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:05.183 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.1.asok 2024-06-03T07:12:05.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69989/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-06-03T07:12:05.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-03T07:12:05.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-03T07:12:05.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-03T07:12:05.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-06-03T07:12:05.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-03T07:12:05.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-03T07:12:05.275 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:12:05.275 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:05.276 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:05.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.1.asok 2024-06-03T07:12:05.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-03T07:12:05.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69989/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-06-03T07:12:05.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-06-03T07:12:05.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-03T07:12:05.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-03T07:12:05.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-06-03T07:12:05.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-03T07:12:05.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-03T07:12:05.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-03T07:12:05.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-06-03T07:12:05.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-03T07:12:05.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-03T07:12:05.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-03T07:12:05.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-06-03T07:12:05.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-03T07:12:05.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-06-03T07:12:05.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-03T07:12:05.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-06-03T07:12:05.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-06-03T07:12:05.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T07:12:05.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-03T07:12:05.661 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-06-03T07:12:05.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-03T07:12:05.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-03T07:12:05.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-03T07:12:05.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-06-03T07:12:05.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-03T07:12:05.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-03T07:12:05.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-06-03T07:12:05.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-03T07:12:05.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-06-03T07:12:05.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-03T07:12:05.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-06-03T07:12:05.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-03T07:12:05.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-03T07:12:05.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-03T07:12:05.664 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-06-03T07:12:05.664 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T07:12:05.664 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T07:12:05.664 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:12:05.664 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:05.664 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:05.664 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.0.asok 2024-06-03T07:12:05.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69989/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-06-03T07:12:05.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-03T07:12:05.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-03T07:12:05.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-03T07:12:05.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-06-03T07:12:05.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T07:12:05.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T07:12:05.756 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:12:05.756 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:05.756 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:05.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.0.asok 2024-06-03T07:12:05.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-03T07:12:05.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69989/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-06-03T07:12:05.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-06-03T07:12:05.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-03T07:12:05.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-03T07:12:05.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-06-03T07:12:05.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-03T07:12:05.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-03T07:12:05.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-03T07:12:05.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-06-03T07:12:05.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-03T07:12:05.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-03T07:12:05.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-03T07:12:05.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-06-03T07:12:05.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-03T07:12:05.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-06-03T07:12:05.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-03T07:12:05.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-06-03T07:12:05.850 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-06-03T07:12:05.850 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T07:12:06.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-03T07:12:06.146 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-06-03T07:12:06.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-03T07:12:06.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-03T07:12:06.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-03T07:12:06.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-06-03T07:12:06.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-03T07:12:06.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-03T07:12:06.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-06-03T07:12:06.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-03T07:12:06.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-06-03T07:12:06.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-03T07:12:06.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-06-03T07:12:06.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-03T07:12:06.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-03T07:12:06.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-03T07:12:06.149 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-06-03T07:12:06.149 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T07:12:06.149 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T07:12:06.149 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:12:06.149 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:06.149 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:06.149 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.0.asok 2024-06-03T07:12:06.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69989/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-06-03T07:12:06.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-03T07:12:06.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-03T07:12:06.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-03T07:12:06.240 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-06-03T07:12:06.240 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T07:12:06.240 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T07:12:06.240 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:12:06.240 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:06.240 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:06.240 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.0.asok 2024-06-03T07:12:06.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-03T07:12:06.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69989/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-06-03T07:12:06.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-06-03T07:12:06.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-03T07:12:06.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-03T07:12:06.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-06-03T07:12:06.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-03T07:12:06.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-03T07:12:06.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-03T07:12:06.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-06-03T07:12:06.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-03T07:12:06.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-03T07:12:06.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-03T07:12:06.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-06-03T07:12:06.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-03T07:12:06.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-06-03T07:12:06.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-03T07:12:06.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-06-03T07:12:06.335 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-06-03T07:12:06.335 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T07:12:06.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-03T07:12:06.625 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-06-03T07:12:06.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-03T07:12:06.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-03T07:12:06.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-03T07:12:06.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-06-03T07:12:06.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-03T07:12:06.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-03T07:12:06.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-06-03T07:12:06.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-03T07:12:06.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-06-03T07:12:06.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-03T07:12:06.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-03T07:12:06.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-03T07:12:06.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-03T07:12:06.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-03T07:12:06.627 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-03T07:12:06.627 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-03T07:12:06.627 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-03T07:12:06.628 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:12:06.628 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:06.628 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:06.628 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.1.asok 2024-06-03T07:12:06.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69989/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-06-03T07:12:06.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-03T07:12:06.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-03T07:12:06.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-03T07:12:06.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-06-03T07:12:06.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-03T07:12:06.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-03T07:12:06.719 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:12:06.719 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:06.719 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:06.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-osd.1.asok 2024-06-03T07:12:06.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-03T07:12:06.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69989/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-06-03T07:12:06.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-06-03T07:12:06.911 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-06-03T07:12:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0: osd_max_object_size = '' 2024-06-03T07:12:06.928 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-06-03T07:12:06.928 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1: osd_max_object_size = '' 2024-06-03T07:12:06.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-06-03T07:12:06.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-06-03T07:12:06.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:12:06.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:12:06.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:12:06.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:12:06.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:12:07.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-06-03T07:07:24.364147+0000 2024-06-03T07:12:07.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-06-03T07:12:07.511 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-06-03T07:12:07.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-06-03T07:07:24.364147+0000 last_deep_scrub_stamp 2024-06-03T07:12:07.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-03T07:12:07.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T07:07:24.364147+0000 2024-06-03T07:12:07.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-06-03T07:12:07.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:12:07.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:12:07.524 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:12:07.524 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:12:07.524 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:12:07.524 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:12:07.524 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:12:07.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:07:24.364147+0000 '>' 2024-06-03T07:07:24.364147+0000 2024-06-03T07:12:07.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:12:08.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:12:08.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:12:08.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:12:08.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:12:08.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:12:08.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:12:08.810 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:12:09.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:07:24.364147+0000 '>' 2024-06-03T07:07:24.364147+0000 2024-06-03T07:12:09.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:12:10.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:12:10.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:12:10.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:12:10.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:12:10.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:12:10.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:12:10.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:12:10.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:07:24.364147+0000 '>' 2024-06-03T07:07:24.364147+0000 2024-06-03T07:12:10.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:12:11.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:12:11.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:12:11.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:12:11.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:12:11.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:12:11.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:12:11.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:12:11.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:07:24.364147+0000 '>' 2024-06-03T07:07:24.364147+0000 2024-06-03T07:12:11.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:12:12.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:12:12.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:12:12.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:12:12.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:12:12.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:12:12.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:12:12.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:12:13.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:07:24.364147+0000 '>' 2024-06-03T07:07:24.364147+0000 2024-06-03T07:12:13.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:12:14.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:12:14.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:12:14.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-03T07:12:14.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:12:14.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:12:14.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:12:14.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-03T07:12:14.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:12:07.939652+0000 '>' 2024-06-03T07:07:24.364147+0000 2024-06-03T07:12:14.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:12:14.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-06-03T07:12:14.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-06-03T07:12:14.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-06-03T07:12:14.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-06-03T07:12:14.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-06-03T07:12:14.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-06-03T07:12:14.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-06-03T07:12:14.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-06-03T07:12:14.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-06-03T07:12:14.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-06-03T07:12:14.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-06-03T07:12:14.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-06-03T07:12:14.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-06-03T07:12:14.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-06-03T07:12:14.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-06-03T07:12:14.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-06-03T07:12:14.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-06-03T07:12:14.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-06-03T07:12:14.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-06-03T07:12:14.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-06-03T07:12:14.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-06-03T07:12:14.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-06-03T07:12:14.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-06-03T07:12:14.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-06-03T07:12:14.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-06-03T07:12:14.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-06-03T07:12:14.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-06-03T07:12:14.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-06-03T07:12:14.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-06-03T07:12:14.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-06-03T07:12:14.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-06-03T07:12:14.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-06-03T07:12:14.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-06-03T07:12:14.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-06-03T07:12:14.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-06-03T07:12:14.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-06-03T07:12:14.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-06-03T07:12:14.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-06-03T07:12:14.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-06-03T07:12:14.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-06-03T07:12:14.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-03T07:12:14.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-06-03T07:12:14.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-06-03T07:12:14.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-06-03T07:12:14.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-06-03T07:12:14.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-03T07:12:14.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-06-03T07:12:14.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-06-03T07:12:14.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-06-03T07:12:14.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=206 2024-06-03T07:12:14.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-06-03T07:12:14.740 INFO:tasks.workunit.client.0.smithi039.stderr: . as $in 2024-06-03T07:12:14.740 INFO:tasks.workunit.client.0.smithi039.stderr: | if type == "object" then 2024-06-03T07:12:14.740 INFO:tasks.workunit.client.0.smithi039.stderr: reduce keys[] as $key 2024-06-03T07:12:14.740 INFO:tasks.workunit.client.0.smithi039.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-03T07:12:14.740 INFO:tasks.workunit.client.0.smithi039.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-03T07:12:14.741 INFO:tasks.workunit.client.0.smithi039.stderr: else f 2024-06-03T07:12:14.741 INFO:tasks.workunit.client.0.smithi039.stderr: end; 2024-06-03T07:12:14.741 INFO:tasks.workunit.client.0.smithi039.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-03T07:12:14.741 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-03T07:12:14.741 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-03T07:12:14.741 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-03T07:12:14.741 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-03T07:12:14.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-06-03T07:12:14.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-03T07:12:14.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-06-03T07:12:14.765 INFO:tasks.workunit.client.0.smithi039.stderr: . as $in 2024-06-03T07:12:14.765 INFO:tasks.workunit.client.0.smithi039.stderr: | if type == "object" then 2024-06-03T07:12:14.765 INFO:tasks.workunit.client.0.smithi039.stderr: reduce keys[] as $key 2024-06-03T07:12:14.765 INFO:tasks.workunit.client.0.smithi039.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-03T07:12:14.766 INFO:tasks.workunit.client.0.smithi039.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-03T07:12:14.766 INFO:tasks.workunit.client.0.smithi039.stderr: else f 2024-06-03T07:12:14.766 INFO:tasks.workunit.client.0.smithi039.stderr: end; 2024-06-03T07:12:14.766 INFO:tasks.workunit.client.0.smithi039.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-03T07:12:14.766 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-03T07:12:14.766 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-03T07:12:14.766 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-03T07:12:14.766 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-03T07:12:14.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-06-03T07:12:14.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-03T07:12:14.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-03T07:12:14.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-03T07:12:14.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-06-03T07:12:14.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-06-03T07:12:14.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-06-03T07:12:14.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=3.0 2024-06-03T07:12:14.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 3.0 2024-06-03T07:12:14.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:12:14.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:12:14.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:12:14.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:12:15.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T07:12:07.939652+0000 2024-06-03T07:12:15.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 3.0 2024-06-03T07:12:15.348 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 3.0 on osd.1 to repair 2024-06-03T07:12:15.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 3.0 2024-06-03T07:12:07.939652+0000 2024-06-03T07:12:15.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-03T07:12:15.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T07:12:07.939652+0000 2024-06-03T07:12:15.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T07:12:15.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:12:15.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:12:15.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:12:15.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:12:15.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:12:15.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:12:15.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:12:15.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:12:07.939652+0000 '>' 2024-06-03T07:12:07.939652+0000 2024-06-03T07:12:15.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:12:16.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:12:16.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:12:16.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:12:16.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:12:16.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:12:16.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:12:16.648 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:12:16.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:12:07.939652+0000 '>' 2024-06-03T07:12:07.939652+0000 2024-06-03T07:12:16.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:12:17.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:12:17.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:12:17.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:12:17.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:12:17.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:12:17.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:12:17.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:12:18.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:12:16.615595+0000 '>' 2024-06-03T07:12:07.939652+0000 2024-06-03T07:12:18.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:12:18.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-06-03T07:12:18.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:12:18.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:12:18.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:12:18.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:12:18.249 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:12:18.249 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:12:18.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:12:18.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:12:18.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:12:18.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:12:18.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:12:18.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:12:18.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:12:18.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:12:18.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:12:18.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:12:18.639 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:12:18.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:12:18.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:12:18.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:12:18.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=858993459209 2024-06-03T07:12:18.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 858993459209 2024-06-03T07:12:18.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-858993459209' 2024-06-03T07:12:18.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:12:18.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:12:18.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=884763262982 2024-06-03T07:12:18.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 884763262982 2024-06-03T07:12:18.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-858993459209 1-884763262982' 2024-06-03T07:12:18.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:12:18.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-858993459209 2024-06-03T07:12:18.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:12:18.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:12:18.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-858993459209 2024-06-03T07:12:18.859 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:12:18.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=858993459209 2024-06-03T07:12:18.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 858993459209' 2024-06-03T07:12:18.859 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 858993459209 2024-06-03T07:12:18.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:12:19.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 858993459207 -lt 858993459209 2024-06-03T07:12:19.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:12:20.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:12:20.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:12:20.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 858993459209 -lt 858993459209 2024-06-03T07:12:20.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:12:20.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-884763262982 2024-06-03T07:12:20.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:12:20.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:12:20.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-884763262982 2024-06-03T07:12:20.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:12:20.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=884763262982 2024-06-03T07:12:20.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 884763262982' 2024-06-03T07:12:20.462 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 884763262982 2024-06-03T07:12:20.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:12:20.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 884763262982 -lt 884763262982 2024-06-03T07:12:20.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:12:20.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:12:20.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:12:21.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:12:21.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:12:21.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:12:21.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:12:21.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:12:21.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:12:21.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:12:21.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:12:21.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:12:21.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:12:21.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:12:21.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:12:21.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:12:21.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:12:21.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:12:21.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-06-03T07:12:21.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-06-03T07:12:21.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-06-03T07:12:21.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-06-03T07:12:21.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-06-03T07:12:21.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-06-03T07:12:21.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-06-03T07:12:21.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-06-03T07:12:22.362 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'csr_pool' removed 2024-06-03T07:12:22.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T07:12:22.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:12:22.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:12:22.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:12:22.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:12:22.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:12:22.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:12:22.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:12:22.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:12:22.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:12:22.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:12:22.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:12:22.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:12:22.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:12:22.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:12:22.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:12:22.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:12:22.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:12:22.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:12:22.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:12:22.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:12:22.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:12:22.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:12:22.524 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:12:22.524 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:22.524 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:22.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:12:22.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:12:22.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:12:22.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T07:12:22.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T07:12:22.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T07:12:22.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T07:12:22.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:12:22.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:12:22.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:12:22.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:12:22.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:12:22.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:12:22.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:12:22.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:12:22.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:12:22.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:12:22.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:12:22.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:12:22.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:12:22.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:12:22.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:12:22.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:12:22.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:12:22.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:12:22.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:12:22.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:12:22.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:12:22.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:12:22.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:12:22.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:22.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:22.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:12:22.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:12:22.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:12:22.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T07:12:22.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T07:12:22.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:22.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:22.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T07:12:22.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T07:12:22.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T07:12:22.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T07:12:22.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T07:12:22.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-06-03T07:12:22.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5931: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-06-03T07:12:22.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-06-03T07:12:22.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5933: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-06-03T07:12:22.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-06-03T07:12:22.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T07:12:22.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T07:12:22.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T07:12:22.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T07:12:22.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T07:12:22.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-06-03T07:12:22.929 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T07:12:22.929 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:12:22.929 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:12:22.930 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:12:22.930 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:22.930 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:22.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:12:22.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-03T07:12:23.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T07:12:23.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T07:12:23.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:12:23.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:12:23.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T07:12:23.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T07:12:23.115 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:12:23.115 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:12:23.115 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:12:23.116 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:12:23.116 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:23.116 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:23.116 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:12:23.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:12:23.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T07:12:23.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T07:12:23.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:12:23.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:12:23.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T07:12:23.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T07:12:23.201 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:12:23.201 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:12:23.201 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:12:23.202 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:12:23.202 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:23.202 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:23.202 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:12:23.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:12:23.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T07:12:23.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5936: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-06-03T07:12:23.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T07:12:23.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T07:12:23.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T07:12:23.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T07:12:23.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T07:12:23.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T07:12:24.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T07:12:24.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:12:24.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:12:24.098 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:12:24.098 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:24.099 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:24.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:12:24.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:12:24.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:12:24.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5937: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-06-03T07:12:24.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:12:24.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:12:24.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T07:12:24.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:12:24.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:12:24.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:12:24.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:12:24.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:12:24.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:12:24.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:12:24.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:12:24.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:12:24.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:12:24.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:12:24.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:12:24.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:12:24.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:12:24.132 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:12:24.132 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:24.132 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:24.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:12:24.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:12:24.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:12:24.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:12:24.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:12:24.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:12:24.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:12:24.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:12:24.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:12:24.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:12:24.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:12:24.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:12:24.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T07:12:24.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:12:24.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:12:24.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e133a845-64cf-49fc-be2b-6df15cddadb3 2024-06-03T07:12:24.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e133a845-64cf-49fc-be2b-6df15cddadb3' 2024-06-03T07:12:24.135 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 e133a845-64cf-49fc-be2b-6df15cddadb3 2024-06-03T07:12:24.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:12:24.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDYbF1mXWAJCRAA1LjP805atkvDiZwnBKwz4Q== 2024-06-03T07:12:24.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDYbF1mXWAJCRAA1LjP805atkvDiZwnBKwz4Q=="}' 2024-06-03T07:12:24.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e133a845-64cf-49fc-be2b-6df15cddadb3 -i td/osd-scrub-repair/0/new.json 2024-06-03T07:12:24.363 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:12:24.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T07:12:24.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDYbF1mXWAJCRAA1LjP805atkvDiZwnBKwz4Q== --osd-uuid e133a845-64cf-49fc-be2b-6df15cddadb3 2024-06-03T07:12:24.404 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:24.404+0000 7f3312799640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:12:24.414 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:24.414+0000 7f3312799640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:12:24.415 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:24.415+0000 7f3312799640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:12:24.415 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:24.416+0000 7f3312799640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:12:24.415 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:24.416+0000 7f3312799640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:12:24.416 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:24.416+0000 7f3312799640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:12:24.506 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:24.507+0000 7f3312799640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T07:12:30.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T07:12:30.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:12:30.106 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T07:12:30.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T07:12:30.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:12:30.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T07:12:30.491 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T07:12:30.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:12:30.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:12:30.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:12:30.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:12:30.519 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:30.519+0000 7fedfe811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:12:30.521 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:30.521+0000 7fedfe811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:12:30.522 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:30.523+0000 7fedfe811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:12:30.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T07:12:30.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:12:30.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:12:30.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:12:30.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:12:30.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:30.786 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:12:30.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:12:30.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:30.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:31.037 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:31.037+0000 7fedfe811640 -1 Falling back to public interface 2024-06-03T07:12:31.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:32.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:32.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:32.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:12:32.078 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:12:32.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:32.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:32.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:33.367 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:33.367+0000 7fedfe811640 -1 osd.0 0 log_to_monitors true 2024-06-03T07:12:33.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:33.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:33.377 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:12:33.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:12:33.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:33.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:33.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:34.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:34.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:34.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:12:34.801 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:12:34.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:34.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:35.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:36.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:36.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:36.132 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:12:36.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:12:36.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:36.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:36.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:37.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:37.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:37.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:12:37.438 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:12:37.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:37.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:37.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:38.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:38.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:38.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:12:38.741 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:12:38.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:38.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:39.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:40.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:40.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:40.129 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:12:40.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:12:40.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:40.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:40.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:41.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:41.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:41.431 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:12:41.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:12:41.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:41.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:41.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:42.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:42.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:42.736 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:12:42.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:12:42.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:42.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:43.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:44.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:44.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:44.116 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:12:44.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:12:44.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:44.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:44.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:45.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:45.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:45.418 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:12:45.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:12:45.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:45.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:45.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:46.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:46.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:46.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:12:46.913 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:12:46.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:46.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:47.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:48.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:48.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:48.602 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:12:48.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:12:48.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:48.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:49.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:50.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:50.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:50.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:12:50.106 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:12:50.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:50.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:50.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:51.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:51.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:51.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:12:51.893 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:12:51.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:51.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:52.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:53.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:53.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:53.383 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:12:53.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:12:53.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:53.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:54.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:12:55.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:12:55.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:12:55.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:12:55.119 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:12:55.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:12:55.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:12:55.551 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3028584668,v1:127.0.0.1:6803/3028584668] [v2:127.0.0.1:6804/3028584668,v1:127.0.0.1:6805/3028584668] exists,up e133a845-64cf-49fc-be2b-6df15cddadb3 2024-06-03T07:12:55.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:12:55.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:12:55.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:12:55.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5938: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-06-03T07:12:55.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:12:55.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:12:55.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T07:12:55.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:12:55.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:12:55.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:12:55.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:12:55.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:12:55.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:12:55.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:12:55.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:12:55.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:12:55.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:12:55.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:12:55.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:12:55.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:12:55.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:12:55.557 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:12:55.557 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:12:55.557 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:12:55.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:12:55.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:12:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:12:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:12:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:12:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:12:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:12:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:12:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:12:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:12:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:12:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:12:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T07:12:55.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:12:55.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:12:55.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ef17805a-4b29-4ede-bef3-e74d070839e7 2024-06-03T07:12:55.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ef17805a-4b29-4ede-bef3-e74d070839e7' 2024-06-03T07:12:55.559 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 ef17805a-4b29-4ede-bef3-e74d070839e7 2024-06-03T07:12:55.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:12:55.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD3bF1mdOcrIhAAZq0t8o9jP7KRMTDnrDZYTQ== 2024-06-03T07:12:55.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD3bF1mdOcrIhAAZq0t8o9jP7KRMTDnrDZYTQ=="}' 2024-06-03T07:12:55.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ef17805a-4b29-4ede-bef3-e74d070839e7 -i td/osd-scrub-repair/1/new.json 2024-06-03T07:12:55.918 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:12:55.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T07:12:55.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD3bF1mdOcrIhAAZq0t8o9jP7KRMTDnrDZYTQ== --osd-uuid ef17805a-4b29-4ede-bef3-e74d070839e7 2024-06-03T07:12:55.957 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:55.957+0000 7f4011580640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:12:56.000 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:56.000+0000 7f4011580640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:12:56.000 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:56.001+0000 7f4011580640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:12:56.001 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:56.001+0000 7f4011580640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:12:56.001 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:56.002+0000 7f4011580640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:12:56.001 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:56.002+0000 7f4011580640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:12:56.066 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:12:56.067+0000 7f4011580640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T07:13:02.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T07:13:02.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:13:02.110 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T07:13:02.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T07:13:02.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:13:02.486 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T07:13:02.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T07:13:02.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:13:02.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:13:02.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:13:02.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:13:02.514 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:13:02.514+0000 7fda0e318640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:13:02.516 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:13:02.517+0000 7fda0e318640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:13:02.517 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:13:02.518+0000 7fda0e318640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:13:02.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T07:13:02.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:13:02.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:13:02.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:13:02.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:13:02.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:02.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:13:02.776 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:13:02.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:02.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:03.034 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:13:03.035+0000 7fda0e318640 -1 Falling back to public interface 2024-06-03T07:13:03.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:04.069 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:13:04.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:04.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:04.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:13:04.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:04.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:04.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:05.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:05.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:05.368 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:13:05.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:13:05.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:05.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:05.460 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:13:05.461+0000 7fda0e318640 -1 osd.1 0 log_to_monitors true 2024-06-03T07:13:05.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:06.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:06.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:06.678 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:13:06.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:13:06.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:06.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:06.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:07.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:07.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:07.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:13:07.982 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:13:07.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:07.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:08.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:09.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:09.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:09.324 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:13:09.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:13:09.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:09.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:09.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:10.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:10.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:10.629 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:13:10.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:13:10.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:10.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:10.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:11.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:11.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:11.938 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:13:11.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:13:11.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:11.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:12.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:13.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:13.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:13.326 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:13:13.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:13:13.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:13.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:13.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:14.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:14.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:14.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:13:14.656 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:13:14.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:14.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:14.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:15.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:15.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:15.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:13:15.960 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:13:15.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:15.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:16.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:17.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:17.342 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:13:17.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:17.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:13:17.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:17.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:17.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:18.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:18.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:18.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:13:18.662 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:13:18.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:18.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:19.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:20.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:20.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:20.515 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:13:20.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:13:20.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:20.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:21.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:22.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:22.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:22.193 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:13:22.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:13:22.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:22.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:22.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:23.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:23.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:23.614 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:13:23.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:13:23.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:23.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:24.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:25.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:25.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:25.364 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:13:25.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:13:25.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:25.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:25.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:26.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:26.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:26.854 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:13:26.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:13:26.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:26.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:27.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:28.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:28.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:28.621 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T07:13:28.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T07:13:28.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:28.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:28.924 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3519498684,v1:127.0.0.1:6811/3519498684] [v2:127.0.0.1:6812/3519498684,v1:127.0.0.1:6813/3519498684] exists,up ef17805a-4b29-4ede-bef3-e74d070839e7 2024-06-03T07:13:28.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:13:28.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:13:28.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:13:28.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5939: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-06-03T07:13:28.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T07:13:29.229 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-06-03T07:13:29.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T07:13:29.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T07:13:29.639 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T07:13:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:13:30.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T07:13:32.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5940: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-06-03T07:13:32.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:13:32.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:13:32.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:13:32.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:13:32.133 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:13:32.133 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:13:32.133 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:13:32.133 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:13:32.133 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:13:32.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:13:32.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:13:32.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:13:32.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:13:32.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:13:32.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:13:32.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:13:32.538 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:13:32.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:13:32.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:13:32.538 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:13:32.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-06-03T07:13:32.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-06-03T07:13:32.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-06-03T07:13:32.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:13:32.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:13:32.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-06-03T07:13:32.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-06-03T07:13:32.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2024-06-03T07:13:32.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:13:32.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-06-03T07:13:32.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:13:32.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:13:32.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-06-03T07:13:32.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:13:32.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-06-03T07:13:32.761 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836491 2024-06-03T07:13:32.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-06-03T07:13:32.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:13:33.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836491 2024-06-03T07:13:33.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:13:34.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:13:34.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:13:34.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836491 2024-06-03T07:13:34.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:13:35.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T07:13:35.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:13:35.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836491 2024-06-03T07:13:35.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:13:35.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-06-03T07:13:35.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:13:35.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:13:35.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-06-03T07:13:35.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:13:35.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-06-03T07:13:35.655 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672970 2024-06-03T07:13:35.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-06-03T07:13:35.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:13:35.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672970 2024-06-03T07:13:35.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:13:35.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:13:35.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:13:36.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T07:13:36.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:13:36.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:13:36.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:13:36.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:13:36.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:13:36.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:13:36.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:13:36.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T07:13:36.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:13:36.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:13:36.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:13:36.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T07:13:36.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:13:36.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:13:36.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-06-03T07:13:36.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-06-03T07:13:37.485 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' created 2024-06-03T07:13:37.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:13:38.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-06-03T07:13:38.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-06-03T07:13:38.918 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'csr_pool' created 2024-06-03T07:13:38.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:13:39.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5944: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-06-03T07:13:39.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:13:39.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:13:39.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:13:39.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:13:39.933 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:13:39.933 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:13:39.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:13:39.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:13:39.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:13:40.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:13:40.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:13:40.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:13:40.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:13:40.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:13:40.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:13:40.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:13:40.339 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:13:40.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:13:40.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:13:40.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:13:40.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-06-03T07:13:40.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-06-03T07:13:40.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-06-03T07:13:40.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:13:40.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:13:40.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-06-03T07:13:40.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-06-03T07:13:40.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672973' 2024-06-03T07:13:40.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:13:40.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-06-03T07:13:40.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:13:40.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:13:40.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-06-03T07:13:40.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:13:40.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-06-03T07:13:40.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-06-03T07:13:40.575 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836493 2024-06-03T07:13:40.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:13:40.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-06-03T07:13:40.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:13:41.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:13:41.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:13:42.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-06-03T07:13:42.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:13:43.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T07:13:43.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:13:43.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836493 2024-06-03T07:13:43.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:13:43.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-06-03T07:13:43.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:13:43.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:13:43.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-06-03T07:13:43.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:13:43.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-06-03T07:13:43.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-06-03T07:13:43.506 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672973 2024-06-03T07:13:43.507 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:13:43.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-06-03T07:13:43.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:13:43.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:13:43.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:13:44.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:13:44.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:13:44.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:13:44.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:13:44.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:13:44.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:13:44.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:13:44.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:13:44.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:13:44.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:13:44.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:13:44.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:13:44.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:13:44.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:13:44.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:13:44.821 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-06-03T07:13:44.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-06-03T07:13:44.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-06-03T07:13:44.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-06-03T07:13:44.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:13:44.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:13:44.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-06-03T07:13:44.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:13:44.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:13:44.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:13:45.207 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:13:45.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:13:45.634 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:13:45.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:13:45.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:13:45.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:13:45.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-06-03T07:13:46.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-06-03T07:13:46.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-06-03T07:13:46.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-06-03T07:13:46.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-06-03T07:13:46.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:13:46.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-03T07:13:46.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-06-03T07:13:46.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:13:46.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:13:46.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:13:46.630 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:13:46.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:13:47.023 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:13:47.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:13:47.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:13:47.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:13:47.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-06-03T07:13:47.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-06-03T07:13:47.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-06-03T07:13:47.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-06-03T07:13:47.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-06-03T07:13:47.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-06-03T07:13:47.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T07:13:47.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-06-03T07:13:47.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-06-03T07:13:47.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-06-03T07:13:47.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-06-03T07:13:47.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-06-03T07:13:47.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T07:13:48.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-06-03T07:13:48.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-06-03T07:13:48.411 INFO:tasks.workunit.client.0.smithi039.stdout:created pool csr_pool snap snap1 2024-06-03T07:13:48.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-06-03T07:13:48.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-06-03T07:13:48.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-06-03T07:13:48.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-06-03T07:13:48.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-06-03T07:13:48.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-06-03T07:13:48.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-06-03T07:13:48.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5968: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-06-03T07:13:48.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-06-03T07:13:48.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:13:48.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:13:48.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-03T07:13:48.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:13:48.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-06-03T07:13:48.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:13:48.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:13:48.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-03T07:13:48.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:13:48.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:13:48.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:13:48.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:13:48.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:13:48.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:13:48.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:13:48.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:13:48.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-06-03T07:13:48.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:13:48.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:13:48.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:13:48.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:13:48.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:13:48.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-06-03T07:13:50.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-03T07:13:50.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:13:50.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:13:50.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:13:50.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:13:50.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:13:50.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:13:50.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:13:50.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:13:50.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:13:50.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:13:50.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:13:50.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:13:50.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:13:50.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:13:50.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:13:50.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:13:50.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:13:50.982 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:13:50.983 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:13:50.983 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:13:50.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:13:50.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:13:50.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:13:50.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:13:50.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:13:50.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:13:50.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:13:50.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:13:50.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:13:50.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:13:50.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:13:50.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:13:50.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:13:50.985 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:13:50.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:13:50.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:13:50.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:13:50.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:13:50.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:13:50.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:13:51.012 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:13:51.013+0000 7fa4ef898640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:13:51.015 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:13:51.016+0000 7fa4ef898640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:13:51.017 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:13:51.017+0000 7fa4ef898640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:13:51.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:13:51.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:13:51.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:13:51.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:13:51.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:13:51.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:51.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:13:51.296 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:13:51.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:51.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:51.532 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:13:51.533+0000 7fa4ef898640 -1 Falling back to public interface 2024-06-03T07:13:51.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:52.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:52.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:52.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:13:52.582 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:13:52.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:52.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:52.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:53.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:53.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:53.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:13:53.882 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:13:53.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:53.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:53.996 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:13:53.997+0000 7fa4ef898640 -1 osd.1 28 log_to_monitors true 2024-06-03T07:13:54.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:13:55.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:13:55.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:13:55.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:13:55.231 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:13:55.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:13:55.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:13:55.611 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/3755723130,v1:127.0.0.1:6811/3755723130] [v2:127.0.0.1:6812/3755723130,v1:127.0.0.1:6813/3755723130] exists,up ef17805a-4b29-4ede-bef3-e74d070839e7 2024-06-03T07:13:55.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:13:55.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:13:55.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:13:55.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:13:55.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:13:55.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:13:55.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:13:55.611 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:13:55.611 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:13:55.611 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:13:55.611 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:13:55.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:13:55.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:13:55.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:13:55.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:13:55.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:13:55.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:13:55.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:13:55.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:13:56.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:13:56.001 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:13:56.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:13:56.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:13:56.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:13:56.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-06-03T07:13:56.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-06-03T07:13:56.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-06-03T07:13:56.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:13:56.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:13:56.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-06-03T07:13:56.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-06-03T07:13:56.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-137438953474' 2024-06-03T07:13:56.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:13:56.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-06-03T07:13:56.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:13:56.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:13:56.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-06-03T07:13:56.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:13:56.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-06-03T07:13:56.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-06-03T07:13:56.221 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 21474836498 2024-06-03T07:13:56.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:13:56.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-06-03T07:13:56.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:13:56.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-06-03T07:13:56.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:13:56.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:13:56.525 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-06-03T07:13:56.525 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:13:56.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-06-03T07:13:56.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-06-03T07:13:56.526 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 137438953474 2024-06-03T07:13:56.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:13:56.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-06-03T07:13:56.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:13:56.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:13:56.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:13:57.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:13:57.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:13:57.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:13:57.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:13:57.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:13:57.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:13:57.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:13:57.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:13:57.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:13:57.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:13:57.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:13:57.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:13:57.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:13:57.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:13:57.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:13:57.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-06-03T07:13:57.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-06-03T07:13:57.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-06-03T07:13:57.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-06-03T07:13:57.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-06-03T07:13:57.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5973: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-06-03T07:13:58.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5974: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-06-03T07:13:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:13:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:13:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:13:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:13:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-06-03T07:13:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:13:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:13:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:13:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:13:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:13:58.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:13:58.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:13:58.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:13:58.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:13:58.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:13:58.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:13:58.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-06-03T07:13:58.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:13:58.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:13:58.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:13:58.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:13:58.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:13:58.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-06-03T07:14:00.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-03T07:14:00.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:14:00.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:14:00.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:14:00.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:14:00.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:14:00.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:14:00.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:14:00.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:14:00.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:14:00.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:14:00.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:14:00.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:14:00.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:14:00.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:14:00.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:14:00.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:14:00.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:14:00.194 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:14:00.194 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:14:00.194 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:14:00.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:14:00.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:14:00.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:14:00.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:14:00.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:14:00.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:14:00.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:14:00.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:14:00.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:14:00.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:14:00.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:14:00.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:14:00.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:14:00.197 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:14:00.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:14:00.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:14:00.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:14:00.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:14:00.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:14:00.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:14:00.226 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:00.226+0000 7fe5ca80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:14:00.228 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:00.229+0000 7fe5ca80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:14:00.230 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:00.230+0000 7fe5ca80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:14:00.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:14:00.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:14:00.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:14:00.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:14:00.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:14:00.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:00.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:14:00.501 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:14:00.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:00.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:00.747 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:00.748+0000 7fe5ca80f640 -1 Falling back to public interface 2024-06-03T07:14:00.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:01.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:01.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:01.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:14:01.834 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:14:01.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:01.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:02.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:03.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:03.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:03.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:14:03.136 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:14:03.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:03.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:03.137 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:03.138+0000 7fe5ca80f640 -1 osd.0 34 log_to_monitors true 2024-06-03T07:14:03.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:04.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:04.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:04.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:14:04.441 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:14:04.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:04.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:04.789 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [5,34) [v2:127.0.0.1:6802/2707017271,v1:127.0.0.1:6803/2707017271] [v2:127.0.0.1:6804/2707017271,v1:127.0.0.1:6805/2707017271] exists,up e133a845-64cf-49fc-be2b-6df15cddadb3 2024-06-03T07:14:04.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:14:04.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:14:04.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:14:04.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:14:04.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:14:04.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:14:04.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:14:04.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:14:04.790 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:14:04.790 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:14:04.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:14:04.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:14:04.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:14:04.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:14:04.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:14:04.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:14:04.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:14:04.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:14:04.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:14:05.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:14:05.183 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:14:05.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:14:05.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:14:05.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:14:05.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757250 2024-06-03T07:14:05.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757250 2024-06-03T07:14:05.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-163208757250' 2024-06-03T07:14:05.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:14:05.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:14:05.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953477 2024-06-03T07:14:05.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953477 2024-06-03T07:14:05.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-163208757250 1-137438953477' 2024-06-03T07:14:05.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:14:05.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-163208757250 2024-06-03T07:14:05.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:14:05.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:14:05.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-163208757250 2024-06-03T07:14:05.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:14:05.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757250 2024-06-03T07:14:05.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 163208757250' 2024-06-03T07:14:05.417 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 163208757250 2024-06-03T07:14:05.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:14:05.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 163208757250 2024-06-03T07:14:05.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:14:06.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:14:06.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:14:07.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757250 -lt 163208757250 2024-06-03T07:14:07.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:14:07.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953477 2024-06-03T07:14:07.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:14:07.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:14:07.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953477 2024-06-03T07:14:07.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:14:07.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953477 2024-06-03T07:14:07.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953477' 2024-06-03T07:14:07.015 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 137438953477 2024-06-03T07:14:07.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:14:07.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953477 -lt 137438953477 2024-06-03T07:14:07.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:14:07.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:14:07.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:14:07.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-03T07:14:07.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:14:07.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:14:07.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:14:07.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:14:07.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:14:07.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:14:07.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:14:07.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-03T07:14:07.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:14:07.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:14:07.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:14:08.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-03T07:14:08.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:14:08.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:14:08.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5979: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-06-03T07:14:08.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-06-03T07:14:08.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-06-03T07:14:08.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-06-03T07:14:08.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:14:08.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:14:08.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:14:08.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:14:08.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-06-03T07:13:38.843339+0000 2024-06-03T07:14:08.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-06-03T07:14:08.903 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 3.0 on osd.1 to scrub 2024-06-03T07:14:08.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-06-03T07:13:38.843339+0000 2024-06-03T07:14:08.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-03T07:14:08.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T07:13:38.843339+0000 2024-06-03T07:14:08.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T07:14:08.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:14:08.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:14:08.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:14:08.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:14:08.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:14:08.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:14:08.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:14:09.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:13:38.843339+0000 '>' 2024-06-03T07:13:38.843339+0000 2024-06-03T07:14:09.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:14:10.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:14:10.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:14:10.206 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:14:10.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:14:10.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:14:10.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:14:10.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:14:10.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:13:38.843339+0000 '>' 2024-06-03T07:13:38.843339+0000 2024-06-03T07:14:10.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:14:11.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:14:11.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:14:11.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:14:11.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:14:11.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:14:11.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:14:11.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:14:11.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:13:38.843339+0000 '>' 2024-06-03T07:13:38.843339+0000 2024-06-03T07:14:11.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:14:12.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:14:12.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:14:12.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:14:12.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:14:12.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:14:12.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:14:12.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:14:13.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:13:38.843339+0000 '>' 2024-06-03T07:13:38.843339+0000 2024-06-03T07:14:13.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:14:14.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:14:14.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:14:14.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:14:14.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:14:14.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:14:14.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:14:14.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:14:14.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:13:38.843339+0000 '>' 2024-06-03T07:13:38.843339+0000 2024-06-03T07:14:14.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:14:15.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:14:15.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:14:15.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:14:15.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:14:15.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:14:15.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:14:15.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:14:15.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:13:38.843339+0000 '>' 2024-06-03T07:13:38.843339+0000 2024-06-03T07:14:15.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:14:16.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:14:16.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:14:16.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-03T07:14:16.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-03T07:14:16.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:14:16.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:14:16.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-03T07:14:17.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:14:09.914754+0000 '>' 2024-06-03T07:13:38.843339+0000 2024-06-03T07:14:17.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:14:17.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5983: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-06-03T07:14:17.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-06-03T07:14:17.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-06-03T07:14:17.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-03T07:14:17.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-06-03T07:14:17.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5989: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-06-03T07:14:17.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-06-03T07:14:17.088 INFO:tasks.workunit.client.0.smithi039.stderr: . as $in 2024-06-03T07:14:17.088 INFO:tasks.workunit.client.0.smithi039.stderr: | if type == "object" then 2024-06-03T07:14:17.088 INFO:tasks.workunit.client.0.smithi039.stderr: reduce keys[] as $key 2024-06-03T07:14:17.088 INFO:tasks.workunit.client.0.smithi039.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-03T07:14:17.088 INFO:tasks.workunit.client.0.smithi039.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-03T07:14:17.088 INFO:tasks.workunit.client.0.smithi039.stderr: else f 2024-06-03T07:14:17.088 INFO:tasks.workunit.client.0.smithi039.stderr: end; 2024-06-03T07:14:17.088 INFO:tasks.workunit.client.0.smithi039.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-03T07:14:17.089 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-03T07:14:17.089 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-03T07:14:17.089 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-03T07:14:17.089 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-03T07:14:17.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-06-03T07:14:17.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-03T07:14:17.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-06-03T07:14:17.111 INFO:tasks.workunit.client.0.smithi039.stderr: . as $in 2024-06-03T07:14:17.111 INFO:tasks.workunit.client.0.smithi039.stderr: | if type == "object" then 2024-06-03T07:14:17.112 INFO:tasks.workunit.client.0.smithi039.stderr: reduce keys[] as $key 2024-06-03T07:14:17.112 INFO:tasks.workunit.client.0.smithi039.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-03T07:14:17.112 INFO:tasks.workunit.client.0.smithi039.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-03T07:14:17.112 INFO:tasks.workunit.client.0.smithi039.stderr: else f 2024-06-03T07:14:17.112 INFO:tasks.workunit.client.0.smithi039.stderr: end; 2024-06-03T07:14:17.112 INFO:tasks.workunit.client.0.smithi039.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-03T07:14:17.112 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-03T07:14:17.112 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-03T07:14:17.112 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-03T07:14:17.112 INFO:tasks.workunit.client.0.smithi039.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-03T07:14:17.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-06-03T07:14:17.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-03T07:14:17.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6158: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-03T07:14:17.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-03T07:14:17.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6159: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-06-03T07:14:17.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6164: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-06-03T07:14:17.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6169: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-06-03T07:14:17.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6170: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-06-03T07:14:17.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6171: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-06-03T07:14:17.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6172: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-06-03T07:14:17.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6173: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-06-03T07:14:17.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6174: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-06-03T07:14:17.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6175: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-06-03T07:14:17.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-06-03T07:14:17.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-03T07:14:17.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-06-03T07:14:17.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-03T07:14:17.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-06-03T07:14:17.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-03T07:14:17.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-06-03T07:14:17.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-03T07:14:17.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-06-03T07:14:17.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-03T07:14:17.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-06-03T07:14:17.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-03T07:14:17.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-06-03T07:14:17.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6187: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-06-03T07:14:17.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6193: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-06-03T07:14:17.539 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'csr_pool' removed 2024-06-03T07:14:17.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T07:14:17.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:14:17.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:14:17.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:14:17.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:14:17.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:14:17.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:14:17.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:14:17.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:14:17.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:14:17.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:14:17.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:14:17.660 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:14:17.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:14:17.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:14:17.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:14:17.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:14:17.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:14:17.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:14:17.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:14:17.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:14:17.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:14:17.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:14:17.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:14:17.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:14:17.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:14:17.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:14:17.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:14:17.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:14:17.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T07:14:17.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T07:14:17.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T07:14:17.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T07:14:17.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:14:17.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:14:17.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:14:17.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:14:17.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:14:17.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:14:17.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:14:17.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:14:17.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:14:17.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:14:17.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:14:17.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:14:17.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:14:17.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:14:17.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:14:17.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:14:17.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:14:17.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:14:17.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:14:17.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:14:17.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:14:17.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:14:17.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:14:17.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:14:17.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:14:17.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:14:17.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:14:17.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:14:17.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T07:14:17.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T07:14:17.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:14:17.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:14:17.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T07:14:17.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T07:14:17.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T07:14:17.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T07:14:17.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T07:14:17.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-06-03T07:14:17.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-06-03T07:14:17.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-06-03T07:14:17.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=false 2024-06-03T07:14:17.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-06-03T07:14:17.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-06-03T07:14:17.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T07:14:17.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T07:14:17.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T07:14:17.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T07:14:17.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T07:14:17.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T07:14:18.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T07:14:18.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:14:18.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:14:18.063 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:14:18.064 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:14:18.064 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:14:18.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:14:18.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T07:14:18.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T07:14:18.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T07:14:18.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:14:18.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:14:18.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T07:14:18.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T07:14:18.259 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:14:18.259 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:14:18.259 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:14:18.259 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:14:18.259 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:14:18.259 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:14:18.260 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:14:18.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:14:18.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T07:14:18.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T07:14:18.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:14:18.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:14:18.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T07:14:18.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T07:14:18.344 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:14:18.344 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:14:18.344 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:14:18.344 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:14:18.345 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:14:18.345 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:14:18.345 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:14:18.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:14:18.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T07:14:18.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-06-03T07:14:18.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T07:14:18.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T07:14:18.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T07:14:18.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T07:14:18.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T07:14:18.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T07:14:19.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T07:14:19.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:14:19.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:14:19.251 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:14:19.251 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:14:19.251 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:14:19.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:14:19.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:14:19.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:14:19.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-06-03T07:14:19.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:14:19.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-06-03T07:14:19.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:14:19.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:14:19.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T07:14:19.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:14:19.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:14:19.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:14:19.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:14:19.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:14:19.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:14:19.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:14:19.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:14:19.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:14:19.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:14:19.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:14:19.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:14:19.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:14:19.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:14:19.286 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:14:19.286 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:14:19.286 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:14:19.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:14:19.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:14:19.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:14:19.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:14:19.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:14:19.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:14:19.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:14:19.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:14:19.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:14:19.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:14:19.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:14:19.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:14:19.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T07:14:19.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:14:19.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:14:19.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9c7474a4-50ab-4019-a6a1-98bffdda03d2 2024-06-03T07:14:19.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9c7474a4-50ab-4019-a6a1-98bffdda03d2' 2024-06-03T07:14:19.288 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 9c7474a4-50ab-4019-a6a1-98bffdda03d2 2024-06-03T07:14:19.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:14:19.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBLbV1mJ0kyEhAAWsOadzNks0yldfsFUtBKWg== 2024-06-03T07:14:19.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBLbV1mJ0kyEhAAWsOadzNks0yldfsFUtBKWg=="}' 2024-06-03T07:14:19.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9c7474a4-50ab-4019-a6a1-98bffdda03d2 -i td/osd-scrub-repair/0/new.json 2024-06-03T07:14:19.557 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:14:19.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T07:14:19.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLbV1mJ0kyEhAAWsOadzNks0yldfsFUtBKWg== --osd-uuid 9c7474a4-50ab-4019-a6a1-98bffdda03d2 2024-06-03T07:14:19.596 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:19.595+0000 7fb537b01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:14:19.607 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:19.606+0000 7fb537b01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:14:19.608 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:19.607+0000 7fb537b01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:14:19.608 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:19.608+0000 7fb537b01640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:14:19.609 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:19.608+0000 7fb537b01640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:14:19.609 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:19.608+0000 7fb537b01640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:14:19.699 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:19.698+0000 7fb537b01640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T07:14:25.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T07:14:25.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:14:25.245 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T07:14:25.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T07:14:25.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:14:25.624 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T07:14:25.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T07:14:25.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:14:25.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:14:25.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:14:25.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:14:25.652 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:25.651+0000 7f1fa6a9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:14:25.654 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:25.654+0000 7f1fa6a9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:14:25.655 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:25.655+0000 7f1fa6a9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:14:25.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T07:14:25.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:14:25.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:14:25.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:14:25.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:14:25.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:25.921 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:14:25.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:14:25.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:25.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:26.169 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:26.169+0000 7f1fa6a9c640 -1 Falling back to public interface 2024-06-03T07:14:26.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:27.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:27.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:27.213 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:14:27.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:14:27.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:27.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:27.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:28.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:28.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:28.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:14:28.519 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:14:28.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:28.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:28.523 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:28.524+0000 7f1fa6a9c640 -1 osd.0 0 log_to_monitors true 2024-06-03T07:14:28.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:29.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:29.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:29.992 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:14:29.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:14:29.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:29.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:30.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:31.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:31.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:31.314 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:14:31.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:14:31.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:31.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:31.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:32.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:32.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:32.618 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:14:32.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:14:32.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:32.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:32.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:33.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:33.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:33.919 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:14:33.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:14:33.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:33.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:34.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:35.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:35.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:35.322 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:14:35.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:14:35.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:35.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:35.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:36.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:36.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:36.625 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:14:36.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:14:36.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:36.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:36.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:37.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:37.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:37.935 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:14:37.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:14:37.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:37.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:38.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:39.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:39.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:39.296 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:14:39.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:14:39.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:39.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:39.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:40.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:40.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:40.596 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:14:40.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:14:40.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:40.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:41.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:42.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:42.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:42.211 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:14:42.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:14:42.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:42.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:42.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:43.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:43.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:43.624 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:14:43.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:14:43.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:43.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:44.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:45.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:45.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:45.310 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:14:45.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:14:45.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:45.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:45.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:46.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:46.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:46.820 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:14:46.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:14:46.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:46.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:47.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:48.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:48.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:48.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:14:48.291 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:14:48.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:48.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:48.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:49.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:49.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:49.641 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:14:49.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:14:49.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:49.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:50.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:14:51.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:14:51.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:51.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T07:14:51.380 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T07:14:51.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:51.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:14:51.677 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/855602857,v1:127.0.0.1:6803/855602857] [v2:127.0.0.1:6804/855602857,v1:127.0.0.1:6805/855602857] exists,up 9c7474a4-50ab-4019-a6a1-98bffdda03d2 2024-06-03T07:14:51.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:14:51.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:14:51.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:14:51.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:14:51.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-06-03T07:14:51.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:14:51.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:14:51.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T07:14:51.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:14:51.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:14:51.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:14:51.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:14:51.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:14:51.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:14:51.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:14:51.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:14:51.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:14:51.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:14:51.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:14:51.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:14:51.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:14:51.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:14:51.682 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:14:51.682 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:14:51.682 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:14:51.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:14:51.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:14:51.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:14:51.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:14:51.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:14:51.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:14:51.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:14:51.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:14:51.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:14:51.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:14:51.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:14:51.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:14:51.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T07:14:51.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:14:51.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:14:51.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a30f4c4e-42e6-4628-82ec-bb6b23cc5b8d 2024-06-03T07:14:51.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a30f4c4e-42e6-4628-82ec-bb6b23cc5b8d' 2024-06-03T07:14:51.684 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 a30f4c4e-42e6-4628-82ec-bb6b23cc5b8d 2024-06-03T07:14:51.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:14:51.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBrbV1mfJzDKRAA+WDzbjHCB3M2CCFS54LYow== 2024-06-03T07:14:51.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBrbV1mfJzDKRAA+WDzbjHCB3M2CCFS54LYow=="}' 2024-06-03T07:14:51.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a30f4c4e-42e6-4628-82ec-bb6b23cc5b8d -i td/osd-scrub-repair/1/new.json 2024-06-03T07:14:52.079 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:14:52.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T07:14:52.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBrbV1mfJzDKRAA+WDzbjHCB3M2CCFS54LYow== --osd-uuid a30f4c4e-42e6-4628-82ec-bb6b23cc5b8d 2024-06-03T07:14:52.120 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:52.121+0000 7fec8b221640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:14:52.163 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:52.164+0000 7fec8b221640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:14:52.164 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:52.165+0000 7fec8b221640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:14:52.165 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:52.165+0000 7fec8b221640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:14:52.165 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:52.165+0000 7fec8b221640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:14:52.165 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:52.165+0000 7fec8b221640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:14:52.257 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:52.257+0000 7fec8b221640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T07:14:58.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T07:14:58.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:14:58.274 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T07:14:58.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T07:14:58.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:14:58.653 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T07:14:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T07:14:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:14:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:14:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:14:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:14:58.681 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:58.681+0000 7f1b33226640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:14:58.683 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:58.683+0000 7f1b33226640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:14:58.684 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:58.684+0000 7f1b33226640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:14:58.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T07:14:58.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:14:58.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:14:58.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:14:58.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:14:58.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:14:58.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:14:58.943 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:14:58.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:14:58.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:14:59.214 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:14:59.212+0000 7f1b33226640 -1 Falling back to public interface 2024-06-03T07:14:59.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:00.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:00.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:00.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:15:00.234 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:15:00.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:00.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:00.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:01.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:01.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:01.537 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:15:01.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:15:01.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:01.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:01.612 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:15:01.613+0000 7f1b33226640 -1 osd.1 0 log_to_monitors true 2024-06-03T07:15:01.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:02.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:02.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:02.843 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:15:02.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:15:02.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:02.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:03.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:04.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:04.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:04.220 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:15:04.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:15:04.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:04.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:04.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:05.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:05.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:05.600 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:15:05.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:15:05.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:05.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:05.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:06.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:06.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:06.901 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:15:06.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:15:06.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:06.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:07.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:08.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:08.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:08.227 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:15:08.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:15:08.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:08.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:08.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:09.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:09.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:09.663 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:15:09.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:15:09.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:09.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:09.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:10.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:10.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:10.964 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:15:10.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:15:10.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:10.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:11.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:12.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:12.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:12.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:15:12.262 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:15:12.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:12.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:12.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:13.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:13.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:13.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:15:13.646 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:15:13.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:13.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:13.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:14.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:14.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:14.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:15:14.949 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:15:14.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:14.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:15.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:16.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:16.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:16.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:15:16.518 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:15:16.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:16.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:17.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:18.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:18.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:18.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:15:18.101 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:15:18.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:18.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:18.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:19.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:19.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:19.845 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:15:19.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:15:19.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:19.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:20.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:21.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:21.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:21.808 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:15:21.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:15:21.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:21.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:22.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:23.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:23.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:23.360 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:15:23.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:15:23.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:23.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:23.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:24.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:24.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:24.990 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T07:15:24.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T07:15:24.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:24.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:15:25.289 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4098861267,v1:127.0.0.1:6811/4098861267] [v2:127.0.0.1:6812/4098861267,v1:127.0.0.1:6813/4098861267] exists,up a30f4c4e-42e6-4628-82ec-bb6b23cc5b8d 2024-06-03T07:15:25.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:15:25.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:15:25.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:15:25.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:15:25.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-06-03T07:15:25.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:15:25.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:15:25.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T07:15:25.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:15:25.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T07:15:25.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:15:25.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:15:25.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:15:25.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:15:25.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T07:15:25.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T07:15:25.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:15:25.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:15:25.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:15:25.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:15:25.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:15:25.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:15:25.295 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:15:25.295 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:15:25.295 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:15:25.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:15:25.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:15:25.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:15:25.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:15:25.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:15:25.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:15:25.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:15:25.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:15:25.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:15:25.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:15:25.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:15:25.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:15:25.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T07:15:25.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T07:15:25.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:15:25.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e6130ade-61e4-472a-a182-337c07b0b84e 2024-06-03T07:15:25.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e6130ade-61e4-472a-a182-337c07b0b84e' 2024-06-03T07:15:25.297 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 e6130ade-61e4-472a-a182-337c07b0b84e 2024-06-03T07:15:25.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:15:25.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCNbV1m332wEhAA1HiqD17ZfB/DBCsBP2CZRQ== 2024-06-03T07:15:25.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCNbV1m332wEhAA1HiqD17ZfB/DBCsBP2CZRQ=="}' 2024-06-03T07:15:25.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e6130ade-61e4-472a-a182-337c07b0b84e -i td/osd-scrub-repair/2/new.json 2024-06-03T07:15:25.673 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:15:25.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-03T07:15:25.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCNbV1m332wEhAA1HiqD17ZfB/DBCsBP2CZRQ== --osd-uuid e6130ade-61e4-472a-a182-337c07b0b84e 2024-06-03T07:15:25.714 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:15:25.715+0000 7f4096232640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:15:25.749 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:15:25.750+0000 7f4096232640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:15:25.750 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:15:25.751+0000 7f4096232640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:15:25.751 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:15:25.751+0000 7f4096232640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T07:15:25.751 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:15:25.751+0000 7f4096232640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T07:15:25.751 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:15:25.751+0000 7f4096232640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T07:15:25.841 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:15:25.841+0000 7f4096232640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T07:15:32.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-03T07:15:32.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:15:32.133 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T07:15:32.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T07:15:32.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:15:32.512 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T07:15:32.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T07:15:32.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:15:32.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:15:32.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:15:32.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:15:32.540 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:15:32.541+0000 7f891a51e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:15:32.542 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:15:32.543+0000 7f891a51e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:15:32.543 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:15:32.544+0000 7f891a51e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:15:32.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T07:15:32.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:15:32.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T07:15:32.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:15:32.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:15:32.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:32.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:15:32.802 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:15:32.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:32.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:33.062 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:15:33.063+0000 7f891a51e640 -1 Falling back to public interface 2024-06-03T07:15:33.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:34.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:34.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:34.097 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:15:34.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:15:34.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:34.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:34.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:35.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:35.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:35.398 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:15:35.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:15:35.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:35.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:35.425 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:15:35.425+0000 7f891a51e640 -1 osd.2 0 log_to_monitors true 2024-06-03T07:15:35.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:36.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:36.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:36.768 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:15:36.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:15:36.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:36.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:37.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:38.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:38.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:38.073 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:15:38.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:15:38.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:38.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:38.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:39.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:39.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:39.446 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:15:39.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:15:39.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:39.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:39.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:40.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:40.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:40.747 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:15:40.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:15:40.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:40.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:41.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:42.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:42.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:42.048 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:15:42.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:15:42.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:42.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:42.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:43.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:43.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:43.353 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:15:43.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:15:43.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:43.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:43.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:44.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:44.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:44.733 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:15:44.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:15:44.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:44.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:45.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:46.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:46.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:46.073 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:15:46.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:15:46.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:46.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:46.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:47.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:47.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:47.497 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:15:47.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:15:47.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:47.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:47.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:48.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:48.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:48.805 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:15:48.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:15:48.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:48.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:49.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:50.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:50.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:50.173 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:15:50.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:15:50.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:50.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:50.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:51.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:51.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:51.724 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:15:51.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:15:51.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:51.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:52.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:53.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:53.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:53.469 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:15:53.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:15:53.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:53.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:53.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:54.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:54.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:54.863 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:15:54.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:15:54.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:54.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:55.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:56.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:56.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:56.447 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:15:56.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:15:56.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:56.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:56.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:57.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:57.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:57.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T07:15:57.846 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T07:15:57.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:57.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:58.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:15:59.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:15:59.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:15:59.478 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T07:15:59.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T07:15:59.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:15:59.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:15:59.875 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1567820167,v1:127.0.0.1:6819/1567820167] [v2:127.0.0.1:6820/1567820167,v1:127.0.0.1:6821/1567820167] exists,up e6130ade-61e4-472a-a182-337c07b0b84e 2024-06-03T07:15:59.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:15:59.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:15:59.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:15:59.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-06-03T07:15:59.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T07:16:00.154 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-06-03T07:16:00.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T07:16:00.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T07:16:00.667 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T07:16:00.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:16:01.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T07:16:03.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-06-03T07:16:03.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:16:03.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:16:03.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:16:03.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:16:03.632 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:16:03.632 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:16:03.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:16:03.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:16:03.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:16:03.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:16:03.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:16:03.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:16:03.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:16:03.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:16:03.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:16:04.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:16:04.033 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:16:04.033 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:16:04.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:16:04.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:16:04.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:16:04.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-06-03T07:16:04.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-06-03T07:16:04.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-06-03T07:16:04.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:16:04.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:16:04.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-06-03T07:16:04.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-06-03T07:16:04.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672972' 2024-06-03T07:16:04.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:16:04.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:16:04.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-06-03T07:16:04.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-06-03T07:16:04.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672972 2-64424509442' 2024-06-03T07:16:04.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:16:04.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-06-03T07:16:04.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:16:04.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:16:04.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-06-03T07:16:04.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:16:04.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-06-03T07:16:04.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-06-03T07:16:04.361 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836497 2024-06-03T07:16:04.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:16:04.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-06-03T07:16:04.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:16:05.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:16:05.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:16:05.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-06-03T07:16:05.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:16:05.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-06-03T07:16:05.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:16:05.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:16:05.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-06-03T07:16:05.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:16:05.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-06-03T07:16:05.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-06-03T07:16:05.954 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672972 2024-06-03T07:16:05.954 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:16:06.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-06-03T07:16:06.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:16:06.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-06-03T07:16:06.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:16:06.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:16:06.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-06-03T07:16:06.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:16:06.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-06-03T07:16:06.245 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509442 2024-06-03T07:16:06.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-06-03T07:16:06.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:16:06.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509443 -lt 64424509442 2024-06-03T07:16:06.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:16:06.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:16:06.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:16:06.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T07:16:06.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:16:06.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:16:06.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:16:06.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:16:06.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:16:06.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:16:06.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:16:07.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T07:16:07.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:16:07.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:16:07.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:16:07.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T07:16:07.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:16:07.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:16:07.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-06-03T07:16:07.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-03T07:16:07.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-03T07:16:07.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-06-03T07:16:07.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-03T07:16:07.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-06-03T07:16:07.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-03T07:16:07.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-03T07:16:08.741 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'ecpool' created 2024-06-03T07:16:08.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:16:09.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-06-03T07:16:09.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-03T07:16:09.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:16:09.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:16:09.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:16:09.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:16:09.756 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:16:09.756 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:16:09.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:16:09.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:16:09.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:16:09.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:16:09.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:16:09.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:16:09.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:16:09.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:16:09.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:16:10.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:16:10.193 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:16:10.193 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:16:10.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:16:10.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:16:10.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:16:10.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-06-03T07:16:10.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-06-03T07:16:10.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-06-03T07:16:10.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:16:10.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:16:10.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-06-03T07:16:10.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-06-03T07:16:10.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672974' 2024-06-03T07:16:10.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:16:10.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:16:10.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-06-03T07:16:10.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-06-03T07:16:10.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672974 2-64424509445' 2024-06-03T07:16:10.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:16:10.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-06-03T07:16:10.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:16:10.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:16:10.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-06-03T07:16:10.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:16:10.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-06-03T07:16:10.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-06-03T07:16:10.531 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836499 2024-06-03T07:16:10.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:16:10.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-06-03T07:16:10.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:16:11.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:16:11.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:16:12.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836499 2024-06-03T07:16:12.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:16:12.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-06-03T07:16:12.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:16:12.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:16:12.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-06-03T07:16:12.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:16:12.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-06-03T07:16:12.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-06-03T07:16:12.165 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672974 2024-06-03T07:16:12.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:16:12.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672974 2024-06-03T07:16:12.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:16:12.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-06-03T07:16:12.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:16:12.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:16:12.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-06-03T07:16:12.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:16:12.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-06-03T07:16:12.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-06-03T07:16:12.472 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509445 2024-06-03T07:16:12.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:16:12.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-06-03T07:16:12.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:16:12.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:16:12.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:16:13.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T07:16:13.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:16:13.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:16:13.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:16:13.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:16:13.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:16:13.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:16:13.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:16:13.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T07:16:13.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:16:13.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:16:13.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:16:13.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-03T07:16:13.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-03T07:16:13.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T07:16:13.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-03T07:16:13.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:16:13.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T07:16:13.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:16:13.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:16:13.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:16:13.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:16:13.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:16:13.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:16:13.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:16:13.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:16:14.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T07:16:14.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:16:14.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:16:14.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:16:14.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-03T07:16:14.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-03T07:16:14.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T07:16:14.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T07:16:14.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T07:16:14.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T07:16:14.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T07:16:14.524 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T07:16:14.524 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T07:16:14.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T07:16:14.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T07:16:14.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T07:16:14.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:16:14.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T07:16:15.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:16:15.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:16:15.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:16:15.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:16:15.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:16:15.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:16:15.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:16:15.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:16:15.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T07:16:15.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:16:15.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:16:15.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:16:15.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-03T07:16:15.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-03T07:16:15.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T07:16:15.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T07:16:15.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T07:16:15.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T07:16:15.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T07:16:15.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T07:16:15.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T07:16:16.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T07:16:16.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T07:16:16.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-03T07:16:16.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T07:16:16.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-03T07:16:16.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T07:16:16.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:16:16.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:16:16.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:16:16.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:16:16.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:16:16.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:16:16.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:16:16.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T07:16:16.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:16:16.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:16:16.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:16:17.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T07:16:17.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:16:17.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:16:17.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-03T07:16:17.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-06-03T07:16:17.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:16:17.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T07:16:17.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-06-03T07:16:17.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:16:17.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:16:17.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:16:17.633 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:16:17.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:16:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:16:18.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:16:18.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:16:18.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:16:18.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-06-03T07:16:18.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-03T07:16:18.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-06-03T07:16:18.425 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-06-03T07:16:18.425 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T07:16:18.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-06-03T07:16:18.791 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:16:18.791 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-06-03T07:16:18.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-06-03T07:16:18.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-06-03T07:16:18.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-06-03T07:16:18.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-06-03T07:16:18.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:16:18.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T07:16:18.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-06-03T07:16:18.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:16:18.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:16:18.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:16:19.174 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:16:19.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:16:19.601 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:16:19.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:16:19.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:16:19.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:16:19.889 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-06-03T07:16:19.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-03T07:16:19.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-06-03T07:16:19.890 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-06-03T07:16:19.890 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T07:16:20.189 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-06-03T07:16:20.189 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:16:20.189 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-06-03T07:16:20.189 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-06-03T07:16:20.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-06-03T07:16:20.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-06-03T07:16:20.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-06-03T07:16:20.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:16:20.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:16:20.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:16:20.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:16:20.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:16:20.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:16:20.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:16:20.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:16:20.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:16:20.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:16:20.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:16:20.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:16:20.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:16:20.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:16:20.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:16:20.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:16:20.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:16:20.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-03T07:16:20.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:16:20.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:16:20.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:16:20.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:16:20.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:16:21.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:16:21.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-06-03T07:16:21.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-06-03T07:16:22.259 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-06-03T07:16:22.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-06-03T07:16:22.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-06-03T07:16:23.913 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-06-03T07:16:24.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-06-03T07:16:24.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-06-03T07:16:25.192 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-06-03T07:16:25.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-06-03T07:16:25.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-06-03T07:16:26.922 INFO:tasks.workunit.client.0.smithi039.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-06-03T07:16:27.438 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-06-03T07:16:27.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:16:27.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-06-03T07:16:27.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:16:27.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:16:27.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:16:27.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:16:27.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:16:27.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:16:27.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:16:27.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:16:27.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:16:27.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:16:27.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:16:27.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:16:27.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:16:27.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:16:27.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:16:27.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:16:27.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:16:27.442 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:16:27.443 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:16:27.443 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:16:27.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:16:27.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:16:27.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:16:27.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:16:27.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:16:27.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:16:27.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:16:27.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:16:27.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:16:27.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:16:27.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:16:27.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:16:27.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:16:27.445 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:16:27.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:16:27.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:16:27.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:16:27.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:16:27.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:16:27.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:16:27.474 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:27.475+0000 7fd0f93fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:16:27.477 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:27.477+0000 7fd0f93fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:16:27.478 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:27.479+0000 7fd0f93fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:16:27.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:16:27.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:16:27.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:16:27.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:16:27.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:16:27.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:16:27.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:16:27.742 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:16:27.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:16:27.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:16:27.992 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:27.993+0000 7fd0f93fc640 -1 Falling back to public interface 2024-06-03T07:16:28.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:16:29.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:16:29.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:16:29.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:16:29.034 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:16:29.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:16:29.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:16:29.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:16:30.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:16:30.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:16:30.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:16:30.336 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:16:30.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:16:30.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:16:30.346 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:30.347+0000 7fd0f93fc640 -1 osd.0 34 log_to_monitors true 2024-06-03T07:16:30.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:16:31.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:16:31.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:16:31.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:16:31.735 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:16:31.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:16:31.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:16:32.040 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 39 up_thru 39 down_at 35 last_clean_interval [5,34) [v2:127.0.0.1:6802/2520017018,v1:127.0.0.1:6803/2520017018] [v2:127.0.0.1:6804/2520017018,v1:127.0.0.1:6805/2520017018] exists,up 9c7474a4-50ab-4019-a6a1-98bffdda03d2 2024-06-03T07:16:32.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:16:32.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:16:32.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:16:32.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:16:32.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-06-03T07:16:32.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:16:32.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:16:32.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:16:32.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:16:32.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:16:32.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:16:32.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:16:32.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:16:32.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:16:32.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:16:32.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:16:32.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:16:32.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:16:32.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:16:32.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:16:32.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:16:32.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:16:32.042 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:16:32.043 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:16:32.043 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:16:32.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:16:32.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:16:32.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:16:32.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:16:32.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:16:32.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:16:32.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:16:32.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:16:32.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:16:32.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:16:32.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:16:32.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:16:32.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:16:32.044 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:16:32.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:16:32.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:16:32.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:16:32.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:16:32.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:16:32.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:16:32.067 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:32.067+0000 7f9c815f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:16:32.069 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:32.070+0000 7f9c815f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:16:32.071 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:32.071+0000 7f9c815f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:16:32.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:16:32.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:16:32.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:16:32.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:16:32.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:16:32.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:16:32.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:16:32.333 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:16:32.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:16:32.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:16:32.587 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:32.588+0000 7f9c815f5640 -1 Falling back to public interface 2024-06-03T07:16:32.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:16:33.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:16:33.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:16:33.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:16:33.627 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:16:33.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:16:33.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:16:33.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:16:34.917 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:34.917+0000 7f9c815f5640 -1 osd.1 35 log_to_monitors true 2024-06-03T07:16:34.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:16:34.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:16:34.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:16:34.927 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:16:34.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:16:34.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:16:35.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:16:36.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:16:36.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:16:36.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:16:36.247 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:16:36.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:16:36.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:16:36.668 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/2230738840,v1:127.0.0.1:6811/2230738840] [v2:127.0.0.1:6812/2230738840,v1:127.0.0.1:6813/2230738840] exists,up a30f4c4e-42e6-4628-82ec-bb6b23cc5b8d 2024-06-03T07:16:36.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:16:36.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:16:36.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:16:36.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:16:36.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-06-03T07:16:36.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:16:36.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:16:36.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-03T07:16:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:16:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T07:16:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:16:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:16:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:16:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:16:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T07:16:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T07:16:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:16:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:16:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:16:36.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:16:36.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:16:36.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:16:36.670 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:16:36.670 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:16:36.670 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:16:36.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:16:36.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:16:36.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:16:36.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:16:36.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:16:36.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:16:36.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:16:36.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:16:36.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:16:36.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:16:36.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:16:36.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T07:16:36.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-03T07:16:36.671 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.2 2024-06-03T07:16:36.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:16:36.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-03T07:16:36.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-03T07:16:36.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:16:36.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:16:36.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:16:36.695 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:36.696+0000 7f7d64c0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:16:36.697 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:36.698+0000 7f7d64c0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:16:36.698 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:36.699+0000 7f7d64c0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:16:36.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-03T07:16:36.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:16:36.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T07:16:36.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:16:36.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:16:36.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:16:36.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:16:36.963 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:16:36.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:16:36.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:16:37.217 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:37.218+0000 7f7d64c0c640 -1 Falling back to public interface 2024-06-03T07:16:37.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:16:38.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:16:38.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:16:38.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:16:38.249 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:16:38.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:16:38.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:16:38.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:16:39.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:16:39.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:16:39.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:16:39.554 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:16:39.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:16:39.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:16:39.693 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:39.693+0000 7f7d64c0c640 -1 osd.2 36 log_to_monitors true 2024-06-03T07:16:39.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:16:40.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:16:40.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:16:40.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:16:40.890 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:16:40.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:16:40.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:16:41.192 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 up in weight 1 up_from 46 up_thru 46 down_at 37 last_clean_interval [15,36) [v2:127.0.0.1:6818/1407314161,v1:127.0.0.1:6819/1407314161] [v2:127.0.0.1:6820/1407314161,v1:127.0.0.1:6821/1407314161] exists,up e6130ade-61e4-472a-a182-337c07b0b84e 2024-06-03T07:16:41.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:16:41.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:16:41.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:16:41.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-06-03T07:16:41.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T07:16:41.629 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' removed 2024-06-03T07:16:41.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T07:16:41.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T07:16:42.031 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T07:16:42.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:16:43.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T07:16:44.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-06-03T07:16:44.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:16:44.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:16:44.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:16:44.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:16:44.841 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:16:44.842 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:16:44.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:16:44.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:16:44.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:16:44.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:16:44.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:16:44.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:16:44.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:16:44.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:16:44.954 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:16:45.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:16:45.248 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:16:45.248 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:16:45.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:16:45.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:16:45.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:16:45.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724548 2024-06-03T07:16:45.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724548 2024-06-03T07:16:45.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724548' 2024-06-03T07:16:45.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:16:45.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:16:45.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626436 2024-06-03T07:16:45.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626436 2024-06-03T07:16:45.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724548 1-180388626436' 2024-06-03T07:16:45.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:16:45.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:16:45.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495619 2024-06-03T07:16:45.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495619 2024-06-03T07:16:45.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724548 1-180388626436 2-197568495619' 2024-06-03T07:16:45.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:16:45.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-167503724548 2024-06-03T07:16:45.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:16:45.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:16:45.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-167503724548 2024-06-03T07:16:45.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:16:45.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724548 2024-06-03T07:16:45.577 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 167503724548 2024-06-03T07:16:45.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 167503724548' 2024-06-03T07:16:45.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:16:45.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724548 -lt 167503724548 2024-06-03T07:16:45.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:16:45.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-180388626436 2024-06-03T07:16:45.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:16:45.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:16:45.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-180388626436 2024-06-03T07:16:45.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:16:45.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626436 2024-06-03T07:16:45.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 180388626436' 2024-06-03T07:16:45.869 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 180388626436 2024-06-03T07:16:45.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:16:46.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626435 -lt 180388626436 2024-06-03T07:16:46.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:16:47.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:16:47.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:16:47.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626436 -lt 180388626436 2024-06-03T07:16:47.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:16:47.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-197568495619 2024-06-03T07:16:47.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:16:47.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:16:47.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-197568495619 2024-06-03T07:16:47.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:16:47.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495619 2024-06-03T07:16:47.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 197568495619' 2024-06-03T07:16:47.513 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 197568495619 2024-06-03T07:16:47.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:16:47.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495619 -lt 197568495619 2024-06-03T07:16:47.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:16:47.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:16:47.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:16:48.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T07:16:48.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:16:48.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:16:48.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:16:48.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:16:48.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:16:48.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:16:48.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:16:48.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T07:16:48.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:16:48.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:16:48.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:16:48.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T07:16:48.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:16:48.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:16:48.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-06-03T07:16:48.859 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T07:16:48.859 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-06-03T07:16:48.859 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-06-03T07:16:48.859 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T07:16:49.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-06-03T07:16:49.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-06-03T07:16:49.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T07:16:49.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T07:16:49.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:16:49.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:16:49.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:16:49.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:16:49.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T07:16:08.623541+0000 2024-06-03T07:16:49.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T07:16:49.773 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-06-03T07:16:49.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T07:16:08.623541+0000 2024-06-03T07:16:49.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T07:16:49.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T07:16:08.623541+0000 2024-06-03T07:16:49.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T07:16:49.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:16:49.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:16:49.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:16:49.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:16:49.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:16:49.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:16:49.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:16:50.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:16:08.623541+0000 '>' 2024-06-03T07:16:08.623541+0000 2024-06-03T07:16:50.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:16:51.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:16:51.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:16:51.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:16:51.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:16:51.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:16:51.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:16:51.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:16:51.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:16:08.623541+0000 '>' 2024-06-03T07:16:08.623541+0000 2024-06-03T07:16:51.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:16:52.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:16:52.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:16:52.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:16:52.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:16:52.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:16:52.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:16:52.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:16:52.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:16:08.623541+0000 '>' 2024-06-03T07:16:08.623541+0000 2024-06-03T07:16:52.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:16:53.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:16:53.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:16:53.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:16:53.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:16:53.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:16:53.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:16:53.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:16:53.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:16:08.623541+0000 '>' 2024-06-03T07:16:08.623541+0000 2024-06-03T07:16:53.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:16:54.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:16:54.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:16:54.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:16:54.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:16:54.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:16:54.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:16:54.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:16:55.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:16:08.623541+0000 '>' 2024-06-03T07:16:08.623541+0000 2024-06-03T07:16:55.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:16:56.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:16:56.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:16:56.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:16:56.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:16:56.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:16:56.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:16:56.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:16:56.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:16:50.678011+0000 '>' 2024-06-03T07:16:08.623541+0000 2024-06-03T07:16:56.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:16:56.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-06-03T07:16:56.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-06-03T07:16:56.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-06-03T07:16:56.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-06-03T07:16:56.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-06-03T07:16:56.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-06-03T07:16:56.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-06-03T07:16:56.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-06-03T07:16:56.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-06-03T07:16:56.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T07:16:56.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:16:56.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:16:56.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:16:56.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:16:56.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:16:56.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:16:56.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:16:56.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:16:56.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:16:56.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:16:56.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:16:56.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:16:56.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:16:56.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:16:56.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:16:56.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:16:56.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:16:56.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:16:56.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:16:56.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:16:56.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:16:56.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:16:56.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:16:56.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:16:56.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:16:56.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:16:56.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:16:56.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:16:56.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T07:16:56.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T07:16:56.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T07:16:56.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T07:16:56.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:16:56.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:16:56.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:16:56.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:16:56.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:16:56.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:16:56.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:16:56.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:16:56.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:16:56.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:16:56.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:16:56.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:16:56.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:16:56.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:16:56.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:16:56.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:16:56.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:16:56.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:16:56.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:16:56.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:16:56.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:16:56.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:16:56.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:16:56.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:16:56.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:16:56.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:16:56.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:16:56.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:16:56.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T07:16:56.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T07:16:56.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:16:56.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:16:56.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T07:16:56.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T07:16:56.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T07:16:56.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T07:16:56.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T07:16:56.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-06-03T07:16:56.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-06-03T07:16:56.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-06-03T07:16:56.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-06-03T07:16:56.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=true 2024-06-03T07:16:56.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-06-03T07:16:56.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-06-03T07:16:56.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T07:16:56.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T07:16:56.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T07:16:56.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T07:16:56.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T07:16:56.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T07:16:57.235 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T07:16:57.235 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:16:57.235 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:16:57.235 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:16:57.235 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:16:57.235 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:16:57.235 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:16:57.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T07:16:57.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T07:16:57.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T07:16:57.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:16:57.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:16:57.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T07:16:57.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T07:16:57.431 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:16:57.431 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:16:57.431 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:16:57.431 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:16:57.431 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:16:57.431 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:16:57.431 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:16:57.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:16:57.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T07:16:57.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T07:16:57.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:16:57.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:16:57.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T07:16:57.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T07:16:57.521 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:16:57.521 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:16:57.521 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:16:57.522 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:16:57.522 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:16:57.522 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:16:57.523 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:16:57.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:16:57.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T07:16:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-06-03T07:16:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T07:16:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T07:16:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T07:16:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T07:16:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T07:16:57.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T07:16:58.422 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T07:16:58.422 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:16:58.422 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:16:58.422 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:16:58.422 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:16:58.423 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:16:58.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:16:58.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:16:58.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:16:58.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-06-03T07:16:58.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:16:58.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-06-03T07:16:58.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:16:58.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:16:58.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T07:16:58.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:16:58.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:16:58.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:16:58.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:16:58.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:16:58.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:16:58.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:16:58.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:16:58.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:16:58.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:16:58.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:16:58.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:16:58.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:16:58.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:16:58.457 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:16:58.457 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:16:58.458 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:16:58.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:16:58.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:16:58.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:16:58.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:16:58.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:16:58.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:16:58.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:16:58.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:16:58.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:16:58.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:16:58.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:16:58.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:16:58.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T07:16:58.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:16:58.459 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:16:58.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1341e9c7-842d-4a24-8694-e11b961e927e 2024-06-03T07:16:58.460 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 1341e9c7-842d-4a24-8694-e11b961e927e 2024-06-03T07:16:58.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1341e9c7-842d-4a24-8694-e11b961e927e' 2024-06-03T07:16:58.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:16:58.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDqbV1mKThpHBAAfmXbRF60JfYc3OOFSYlJfA== 2024-06-03T07:16:58.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDqbV1mKThpHBAAfmXbRF60JfYc3OOFSYlJfA=="}' 2024-06-03T07:16:58.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1341e9c7-842d-4a24-8694-e11b961e927e -i td/osd-scrub-repair/0/new.json 2024-06-03T07:16:58.727 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:16:58.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T07:16:58.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDqbV1mKThpHBAAfmXbRF60JfYc3OOFSYlJfA== --osd-uuid 1341e9c7-842d-4a24-8694-e11b961e927e 2024-06-03T07:16:58.768 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:58.768+0000 7f66ae49a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:16:58.778 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:58.778+0000 7f66ae49a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:16:58.779 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:58.779+0000 7f66ae49a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:16:58.780 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:58.779+0000 7f66ae49a640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:16:58.780 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:58.779+0000 7f66ae49a640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:16:58.780 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:58.779+0000 7f66ae49a640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:16:58.870 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:16:58.870+0000 7f66ae49a640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T07:17:04.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T07:17:04.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:17:04.395 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T07:17:04.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T07:17:04.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:17:04.799 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T07:17:04.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T07:17:04.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:17:04.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:17:04.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:17:04.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:17:04.827 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:04.828+0000 7f1d5ab01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:17:04.829 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:04.830+0000 7f1d5ab01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:17:04.830 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:04.831+0000 7f1d5ab01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:17:05.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T07:17:05.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:17:05.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:17:05.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:17:05.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:17:05.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:05.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:17:05.089 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:17:05.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:05.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:05.344 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:05.345+0000 7f1d5ab01640 -1 Falling back to public interface 2024-06-03T07:17:05.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:06.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:06.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:06.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:17:06.377 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:17:06.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:06.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:06.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:07.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:07.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:07.676 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:17:07.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:17:07.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:07.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:07.688 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:07.689+0000 7f1d5ab01640 -1 osd.0 0 log_to_monitors true 2024-06-03T07:17:08.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:09.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:09.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:09.133 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:17:09.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:17:09.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:09.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:09.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:10.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:10.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:10.456 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:17:10.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:17:10.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:10.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:10.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:11.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:11.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:11.757 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:17:11.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:17:11.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:11.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:12.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:13.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:13.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:13.057 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:17:13.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:17:13.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:13.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:13.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:14.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:14.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:14.414 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:17:14.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:17:14.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:14.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:14.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:15.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:15.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:15.716 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:17:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:17:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:16.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:17.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:17.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:17.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:17:17.018 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:17:17.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:17.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:17.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:18.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:18.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:18.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:17:18.319 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:17:18.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:18.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:18.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:19.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:19.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:19.625 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:17:19.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:17:19.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:19.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:20.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:21.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:21.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:21.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:17:21.279 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:17:21.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:21.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:21.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:22.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:22.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:22.673 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:17:22.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:17:22.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:22.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:23.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:24.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:24.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:24.385 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:17:24.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:17:24.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:24.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:24.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:25.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:25.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:25.804 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:17:25.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:17:25.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:25.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:26.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:27.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:27.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:27.516 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:17:27.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:17:27.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:27.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:28.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:29.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:29.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:29.277 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:17:29.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:17:29.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:29.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:17:29.647 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/44765412,v1:127.0.0.1:6803/44765412] [v2:127.0.0.1:6804/44765412,v1:127.0.0.1:6805/44765412] exists,up 1341e9c7-842d-4a24-8694-e11b961e927e 2024-06-03T07:17:29.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:17:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:17:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:17:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:17:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-06-03T07:17:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:17:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:17:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T07:17:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:17:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:17:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:17:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:17:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:17:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:17:29.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:17:29.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:17:29.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:17:29.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:17:29.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:17:29.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:17:29.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:17:29.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:17:29.652 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:17:29.652 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:17:29.652 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:17:29.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:17:29.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:17:29.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:17:29.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:17:29.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:17:29.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:17:29.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:17:29.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:17:29.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:17:29.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:17:29.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:17:29.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:17:29.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T07:17:29.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:17:29.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:17:29.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9689984a-ce8a-476b-8011-aeb8adad2d1b 2024-06-03T07:17:29.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9689984a-ce8a-476b-8011-aeb8adad2d1b' 2024-06-03T07:17:29.655 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 9689984a-ce8a-476b-8011-aeb8adad2d1b 2024-06-03T07:17:29.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:17:29.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJbl1mZFD9JxAA+AZOnE1TR0q3SsOxl2aCow== 2024-06-03T07:17:29.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJbl1mZFD9JxAA+AZOnE1TR0q3SsOxl2aCow=="}' 2024-06-03T07:17:29.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9689984a-ce8a-476b-8011-aeb8adad2d1b -i td/osd-scrub-repair/1/new.json 2024-06-03T07:17:30.063 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:17:30.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T07:17:30.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJbl1mZFD9JxAA+AZOnE1TR0q3SsOxl2aCow== --osd-uuid 9689984a-ce8a-476b-8011-aeb8adad2d1b 2024-06-03T07:17:30.105 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:30.106+0000 7f1570c2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:17:30.166 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:30.167+0000 7f1570c2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:17:30.167 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:30.168+0000 7f1570c2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:17:30.168 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:30.168+0000 7f1570c2c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:17:30.168 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:30.168+0000 7f1570c2c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:17:30.168 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:30.168+0000 7f1570c2c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:17:30.241 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:30.242+0000 7f1570c2c640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T07:17:36.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T07:17:36.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:17:36.394 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T07:17:36.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T07:17:36.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:17:36.775 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T07:17:36.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T07:17:36.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:17:36.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:17:36.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:17:36.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:17:36.802 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:36.803+0000 7f5154a1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:17:36.804 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:36.805+0000 7f5154a1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:17:36.806 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:36.806+0000 7f5154a1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:17:37.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T07:17:37.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:17:37.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:17:37.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:17:37.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:17:37.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:37.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:17:37.066 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:17:37.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:37.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:37.327+0000 7f5154a1d640 -1 Falling back to public interface 2024-06-03T07:17:37.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:38.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:38.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:38.360 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:17:38.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:17:38.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:38.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:38.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:39.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:39.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:39.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:17:39.656 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:17:39.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:39.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:39.739 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:17:39.740+0000 7f5154a1d640 -1 osd.1 0 log_to_monitors true 2024-06-03T07:17:39.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:40.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:40.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:40.958 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:17:40.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:17:40.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:40.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:41.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:42.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:42.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:42.263 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:17:42.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:17:42.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:42.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:42.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:43.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:43.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:43.566 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:17:43.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:17:43.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:43.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:43.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:44.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:44.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:44.871 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:17:44.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:17:44.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:44.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:45.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:46.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:46.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:46.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:17:46.173 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:17:46.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:46.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:46.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:47.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:47.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:47.526 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:17:47.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:17:47.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:47.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:47.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:48.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:48.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:48.906 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:17:48.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:17:48.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:48.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:49.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:50.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:50.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:50.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:17:50.214 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:17:50.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:50.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:50.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:51.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:51.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:51.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:17:51.567 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:17:51.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:51.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:51.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:52.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:52.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:52.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:17:52.872 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:17:52.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:52.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:53.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:54.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:54.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:54.550 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:17:54.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:17:54.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:54.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:55.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:56.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:56.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:56.278 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:17:56.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:17:56.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:56.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:57.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:58.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:58.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:58.038 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:17:58.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:17:58.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:58.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:17:58.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:17:59.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:17:59.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:17:59.706 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:17:59.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:17:59.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:17:59.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:18:00.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:01.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:01.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:01.367 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:18:01.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:18:01.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:01.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:18:02.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:03.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:03.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:03.069 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T07:18:03.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T07:18:03.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:03.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:18:03.372 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2897749743,v1:127.0.0.1:6811/2897749743] [v2:127.0.0.1:6812/2897749743,v1:127.0.0.1:6813/2897749743] exists,up 9689984a-ce8a-476b-8011-aeb8adad2d1b 2024-06-03T07:18:03.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:18:03.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:18:03.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:18:03.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:18:03.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-06-03T07:18:03.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:18:03.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:18:03.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T07:18:03.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:18:03.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T07:18:03.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:18:03.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:18:03.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:18:03.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:18:03.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T07:18:03.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T07:18:03.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:18:03.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:18:03.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:18:03.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:18:03.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:18:03.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:18:03.378 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:18:03.378 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:18:03.378 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:18:03.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:18:03.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:18:03.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:18:03.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:18:03.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:18:03.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:18:03.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:18:03.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:18:03.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:18:03.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:18:03.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:18:03.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:18:03.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T07:18:03.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T07:18:03.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:18:03.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c3b02073-5363-4b3b-857b-780c39b0df1d 2024-06-03T07:18:03.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c3b02073-5363-4b3b-857b-780c39b0df1d' 2024-06-03T07:18:03.380 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 c3b02073-5363-4b3b-857b-780c39b0df1d 2024-06-03T07:18:03.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:18:03.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQArbl1mGa2kFxAAh9kdIUds85bjkU/StUa3dQ== 2024-06-03T07:18:03.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQArbl1mGa2kFxAAh9kdIUds85bjkU/StUa3dQ=="}' 2024-06-03T07:18:03.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c3b02073-5363-4b3b-857b-780c39b0df1d -i td/osd-scrub-repair/2/new.json 2024-06-03T07:18:03.787 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:18:03.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-03T07:18:03.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQArbl1mGa2kFxAAh9kdIUds85bjkU/StUa3dQ== --osd-uuid c3b02073-5363-4b3b-857b-780c39b0df1d 2024-06-03T07:18:03.830 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:18:03.830+0000 7f1d6e610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:18:03.880 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:18:03.881+0000 7f1d6e610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:18:03.881 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:18:03.882+0000 7f1d6e610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:18:03.882 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:18:03.882+0000 7f1d6e610640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T07:18:03.882 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:18:03.882+0000 7f1d6e610640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T07:18:03.882 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:18:03.882+0000 7f1d6e610640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T07:18:03.960 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:18:03.961+0000 7f1d6e610640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T07:18:10.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-03T07:18:10.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:18:10.251 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T07:18:10.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T07:18:10.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:18:10.647 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T07:18:10.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T07:18:10.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:18:10.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:18:10.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:18:10.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:18:10.675 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:18:10.676+0000 7fd6ee42e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:18:10.677 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:18:10.678+0000 7fd6ee42e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:18:10.678 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:18:10.679+0000 7fd6ee42e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:18:10.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T07:18:10.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:18:10.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T07:18:10.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:18:10.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:18:10.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:10.938 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:18:10.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:18:10.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:10.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:11.193 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:18:11.194+0000 7fd6ee42e640 -1 Falling back to public interface 2024-06-03T07:18:11.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:12.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:12.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:12.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:18:12.227 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:18:12.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:12.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:12.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:13.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:13.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:13.535 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:18:13.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:18:13.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:13.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:13.621 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:18:13.622+0000 7fd6ee42e640 -1 osd.2 0 log_to_monitors true 2024-06-03T07:18:13.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:14.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:14.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:14.881 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:18:14.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:18:14.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:14.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:15.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:16.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:16.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:16.181 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:18:16.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:18:16.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:16.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:16.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:17.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:17.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:17.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:18:17.610 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:18:17.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:17.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:17.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:18.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:18.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:18.991 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:18:18.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:18:18.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:18.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:19.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:20.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:20.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:20.293 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:18:20.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:18:20.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:20.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:20.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:21.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:21.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:21.612 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:18:21.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:18:21.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:21.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:21.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:22.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:22.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:22.932 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:18:22.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:18:22.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:22.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:24.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:24.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:24.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:18:24.237 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:18:24.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:24.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:24.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:25.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:25.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:25.562 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:18:25.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:18:25.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:25.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:25.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:26.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:26.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:26.956 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:18:26.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:18:26.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:26.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:28.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:29.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:29.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:29.025 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:18:29.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:18:29.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:29.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:29.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:30.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:30.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:30.727 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:18:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:18:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:30.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:31.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:32.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:32.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:32.105 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:18:32.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:18:32.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:32.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:32.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:33.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:33.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:33.630 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:18:33.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:18:33.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:33.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:33.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:34.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:34.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:34.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:18:34.930 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:18:34.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:34.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:35.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:18:36.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:18:36.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:18:36.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T07:18:36.712 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T07:18:36.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:18:36.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:18:37.078 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1758057341,v1:127.0.0.1:6819/1758057341] [v2:127.0.0.1:6820/1758057341,v1:127.0.0.1:6821/1758057341] exists,up c3b02073-5363-4b3b-857b-780c39b0df1d 2024-06-03T07:18:37.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:18:37.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:18:37.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:18:37.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-06-03T07:18:37.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T07:18:37.363 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-06-03T07:18:37.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T07:18:37.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T07:18:37.831 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T07:18:37.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:18:38.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T07:18:41.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-06-03T07:18:41.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:18:41.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:18:41.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:18:41.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:18:41.122 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:18:41.122 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:18:41.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:18:41.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:18:41.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:18:41.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:18:41.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:18:41.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:18:41.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:18:41.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:18:41.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:18:41.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:18:41.524 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:18:41.525 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:18:41.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:18:41.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:18:41.525 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:18:41.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-06-03T07:18:41.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-06-03T07:18:41.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-06-03T07:18:41.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:18:41.648 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:18:41.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-06-03T07:18:41.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-06-03T07:18:41.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672972' 2024-06-03T07:18:41.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:18:41.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:18:41.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-06-03T07:18:41.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-06-03T07:18:41.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672972 2-64424509445' 2024-06-03T07:18:41.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:18:41.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-06-03T07:18:41.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:18:41.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:18:41.865 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-06-03T07:18:41.865 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:18:41.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-06-03T07:18:41.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-06-03T07:18:41.867 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836499 2024-06-03T07:18:41.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:18:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-06-03T07:18:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:18:43.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:18:43.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:18:43.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-06-03T07:18:43.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:18:43.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-06-03T07:18:43.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:18:43.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:18:43.459 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-06-03T07:18:43.459 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:18:43.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-06-03T07:18:43.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-06-03T07:18:43.460 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672972 2024-06-03T07:18:43.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:18:43.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-06-03T07:18:43.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:18:43.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-06-03T07:18:43.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:18:43.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:18:43.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-06-03T07:18:43.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:18:43.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-06-03T07:18:43.753 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509445 2024-06-03T07:18:43.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-06-03T07:18:43.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:18:44.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509445 2024-06-03T07:18:44.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:18:44.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:18:44.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:18:44.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T07:18:44.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:18:44.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:18:44.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:18:44.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:18:44.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:18:44.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:18:44.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:18:44.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T07:18:44.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:18:44.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:18:44.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:18:45.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T07:18:45.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:18:45.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:18:45.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-06-03T07:18:45.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-03T07:18:45.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-03T07:18:45.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-06-03T07:18:45.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-03T07:18:45.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-06-03T07:18:45.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-03T07:18:45.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-03T07:18:46.070 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'ecpool' created 2024-06-03T07:18:46.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:18:47.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-06-03T07:18:47.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-06-03T07:18:47.508 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 2 allow_ec_overwrites to true 2024-06-03T07:18:47.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-03T07:18:47.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:18:47.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:18:47.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:18:47.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:18:47.521 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:18:47.521 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:18:47.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:18:47.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:18:47.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:18:47.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:18:47.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:18:47.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:18:47.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:18:47.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:18:47.622 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:18:48.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:18:48.040 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:18:48.040 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:18:48.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:18:48.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:18:48.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:18:48.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-06-03T07:18:48.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-06-03T07:18:48.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-06-03T07:18:48.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:18:48.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:18:48.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-06-03T07:18:48.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-06-03T07:18:48.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672975' 2024-06-03T07:18:48.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:18:48.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:18:48.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-06-03T07:18:48.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-06-03T07:18:48.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672975 2-64424509448' 2024-06-03T07:18:48.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:18:48.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-06-03T07:18:48.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:18:48.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:18:48.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-06-03T07:18:48.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:18:48.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-06-03T07:18:48.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-06-03T07:18:48.438 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836501 2024-06-03T07:18:48.438 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:18:48.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-06-03T07:18:48.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:18:48.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-06-03T07:18:48.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:18:48.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:18:48.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-06-03T07:18:48.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:18:48.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-06-03T07:18:48.738 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672975 2024-06-03T07:18:48.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-06-03T07:18:48.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:18:49.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-06-03T07:18:49.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:18:49.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-06-03T07:18:49.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:18:49.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:18:49.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-06-03T07:18:49.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:18:49.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-06-03T07:18:49.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-06-03T07:18:49.067 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509448 2024-06-03T07:18:49.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:18:49.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-06-03T07:18:49.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:18:49.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:18:49.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:18:49.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T07:18:49.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:18:49.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:18:49.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:18:49.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:18:49.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:18:49.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:18:49.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:18:50.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T07:18:50.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:18:50.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:18:50.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:18:50.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T07:18:50.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:18:50.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:18:50.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-03T07:18:50.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-06-03T07:18:50.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:18:50.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T07:18:50.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-06-03T07:18:50.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:18:50.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:18:50.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:18:50.861 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:18:50.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:18:51.397 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:18:51.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:18:51.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:18:51.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:18:51.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-06-03T07:18:51.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-03T07:18:51.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-06-03T07:18:51.653 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-06-03T07:18:51.653 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T07:18:51.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-06-03T07:18:51.945 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:18:51.945 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-06-03T07:18:51.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-06-03T07:18:51.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-06-03T07:18:51.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-06-03T07:18:51.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-06-03T07:18:51.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:18:51.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-03T07:18:51.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-06-03T07:18:51.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-03T07:18:51.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-03T07:18:51.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-03T07:18:52.334 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is set 2024-06-03T07:18:52.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-03T07:18:52.870 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:18:52.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:18:52.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:18:52.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:18:53.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-06-03T07:18:53.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-03T07:18:53.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-06-03T07:18:53.100 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-06-03T07:18:53.100 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T07:18:53.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-06-03T07:18:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:18:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-06-03T07:18:53.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-06-03T07:18:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-06-03T07:18:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-06-03T07:18:53.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-06-03T07:18:53.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:18:53.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:18:53.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:18:53.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:18:53.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:18:53.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:18:53.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:18:53.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:18:53.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:18:53.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:18:53.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:18:53.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:18:53.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:18:53.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:18:53.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:18:55.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:18:55.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:18:55.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-03T07:18:55.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:18:55.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:18:55.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:18:55.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:18:55.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:18:55.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:18:55.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-06-03T07:18:55.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-06-03T07:18:56.356 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-06-03T07:18:56.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-06-03T07:18:56.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-06-03T07:18:58.059 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-06-03T07:18:58.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-06-03T07:18:58.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-06-03T07:18:59.362 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-06-03T07:18:59.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-06-03T07:18:59.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-06-03T07:19:01.033 INFO:tasks.workunit.client.0.smithi039.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-06-03T07:19:01.552 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-06-03T07:19:01.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:19:01.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-06-03T07:19:01.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:19:01.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:19:01.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:19:01.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:19:01.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:19:01.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:19:01.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:19:01.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:19:01.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:19:01.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:19:01.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:19:01.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:19:01.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:19:01.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:19:01.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:19:01.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:19:01.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:19:01.557 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:19:01.558 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:19:01.558 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:19:01.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:19:01.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:19:01.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:19:01.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:19:01.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:19:01.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:19:01.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:19:01.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:19:01.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:19:01.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:19:01.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:19:01.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:19:01.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:19:01.560 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:19:01.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:19:01.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:19:01.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:19:01.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:19:01.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:19:01.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:19:01.588 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:01.589+0000 7f636fb87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:01.626 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:01.627+0000 7f636fb87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:01.628 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:01.628+0000 7f636fb87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:01.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:19:01.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:19:01.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:19:01.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:19:01.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:19:01.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:01.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:19:01.859 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:19:01.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:01.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:02.146 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:02.147+0000 7f636fb87640 -1 Falling back to public interface 2024-06-03T07:19:02.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:03.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:03.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:03.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:19:03.150 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:19:03.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:03.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:03.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:04.268 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:04.269+0000 7f636fb87640 -1 osd.0 35 log_to_monitors true 2024-06-03T07:19:04.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:04.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:04.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:19:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:19:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:04.831 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 42 up_thru 17 down_at 36 last_clean_interval [5,35) [v2:127.0.0.1:6802/4229167847,v1:127.0.0.1:6803/4229167847] [v2:127.0.0.1:6804/4229167847,v1:127.0.0.1:6805/4229167847] exists,up 1341e9c7-842d-4a24-8694-e11b961e927e 2024-06-03T07:19:04.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:19:04.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:19:04.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:19:04.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:19:04.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-06-03T07:19:04.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:19:04.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:19:04.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:19:04.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:19:04.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:19:04.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:19:04.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:19:04.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:19:04.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:19:04.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:19:04.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:19:04.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:19:04.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:19:04.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:19:04.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:19:04.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:19:04.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:19:04.833 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:19:04.833 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:19:04.833 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:19:04.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:19:04.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:19:04.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:19:04.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:19:04.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:19:04.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:19:04.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:19:04.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:19:04.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:19:04.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:19:04.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:19:04.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:19:04.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:19:04.834 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.1 2024-06-03T07:19:04.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:19:04.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:19:04.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:19:04.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:19:04.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:19:04.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:19:04.858 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:04.859+0000 7f4ad1661640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:04.860 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:04.861+0000 7f4ad1661640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:04.861 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:04.862+0000 7f4ad1661640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:05.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:19:05.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:19:05.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:19:05.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:19:05.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:19:05.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:05.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:19:05.124 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:19:05.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:05.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:19:05.379 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:05.380+0000 7f4ad1661640 -1 Falling back to public interface 2024-06-03T07:19:05.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:06.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:06.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:06.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:19:06.430 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:19:06.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:06.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:19:06.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:07.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:07.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:07.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:19:07.730 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:19:07.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:07.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:19:07.750 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:07.751+0000 7f4ad1661640 -1 osd.1 36 log_to_monitors true 2024-06-03T07:19:08.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:09.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:19:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:19:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:09.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:19:09.398 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/650836950,v1:127.0.0.1:6811/650836950] [v2:127.0.0.1:6812/650836950,v1:127.0.0.1:6813/650836950] exists,up 9689984a-ce8a-476b-8011-aeb8adad2d1b 2024-06-03T07:19:09.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:19:09.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:19:09.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:19:09.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-03T07:19:09.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-06-03T07:19:09.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:19:09.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:19:09.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-03T07:19:09.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:19:09.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T07:19:09.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:19:09.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:19:09.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:19:09.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:19:09.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T07:19:09.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T07:19:09.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:19:09.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:19:09.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:19:09.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:19:09.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:19:09.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:19:09.400 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:19:09.400 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:19:09.400 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:19:09.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:19:09.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:19:09.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:19:09.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:19:09.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:19:09.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:19:09.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:19:09.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:19:09.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:19:09.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:19:09.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T07:19:09.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T07:19:09.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-03T07:19:09.401 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.2 2024-06-03T07:19:09.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T07:19:09.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-03T07:19:09.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-03T07:19:09.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:19:09.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:19:09.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:19:09.425 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:09.425+0000 7fcb4b41f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:09.427 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:09.428+0000 7fcb4b41f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:09.428 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:09.429+0000 7fcb4b41f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:09.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-03T07:19:09.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:19:09.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T07:19:09.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:19:09.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:19:09.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:09.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:19:09.725 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:19:09.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:09.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:19:09.942 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:09.943+0000 7fcb4b41f640 -1 Falling back to public interface 2024-06-03T07:19:10.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:11.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:11.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:11.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:19:11.056 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:19:11.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:11.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:19:11.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:12.274 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:12.275+0000 7fcb4b41f640 -1 osd.2 39 log_to_monitors true 2024-06-03T07:19:12.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:12.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:12.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:19:12.358 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:19:12.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:12.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:19:12.679 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 up in weight 1 up_from 49 up_thru 37 down_at 40 last_clean_interval [15,39) [v2:127.0.0.1:6818/677816992,v1:127.0.0.1:6819/677816992] [v2:127.0.0.1:6820/677816992,v1:127.0.0.1:6821/677816992] exists,up c3b02073-5363-4b3b-857b-780c39b0df1d 2024-06-03T07:19:12.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:19:12.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:19:12.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:19:12.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-06-03T07:19:12.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T07:19:13.881 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' removed 2024-06-03T07:19:13.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T07:19:13.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T07:19:14.359 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T07:19:14.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:19:15.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T07:19:17.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-06-03T07:19:17.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:19:17.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:19:17.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:19:17.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:19:17.272 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:19:17.272 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:19:17.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:19:17.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:19:17.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:19:17.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:19:17.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:19:17.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:19:17.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:19:17.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:19:17.383 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:19:17.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:19:17.677 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:19:17.677 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:19:17.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:19:17.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:19:17.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:19:17.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626436 2024-06-03T07:19:17.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626436 2024-06-03T07:19:17.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-180388626436' 2024-06-03T07:19:17.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:19:17.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:19:17.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528323 2024-06-03T07:19:17.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528323 2024-06-03T07:19:17.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-180388626436 1-193273528323' 2024-06-03T07:19:17.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:19:17.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:19:18.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397507 2024-06-03T07:19:18.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397507 2024-06-03T07:19:18.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-180388626436 1-193273528323 2-210453397507' 2024-06-03T07:19:18.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:19:18.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-180388626436 2024-06-03T07:19:18.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:19:18.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:19:18.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-180388626436 2024-06-03T07:19:18.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:19:18.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626436 2024-06-03T07:19:18.028 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 180388626436 2024-06-03T07:19:18.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 180388626436' 2024-06-03T07:19:18.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:19:18.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626435 -lt 180388626436 2024-06-03T07:19:18.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:19:19.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:19:19.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:19:19.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626436 -lt 180388626436 2024-06-03T07:19:19.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:19:19.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528323 2024-06-03T07:19:19.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:19:19.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:19:19.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528323 2024-06-03T07:19:19.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:19:19.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528323 2024-06-03T07:19:19.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528323' 2024-06-03T07:19:19.629 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 193273528323 2024-06-03T07:19:19.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:19:19.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528324 -lt 193273528323 2024-06-03T07:19:19.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:19:19.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-210453397507 2024-06-03T07:19:19.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:19:19.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:19:19.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-210453397507 2024-06-03T07:19:19.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:19:19.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397507 2024-06-03T07:19:19.923 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 210453397507 2024-06-03T07:19:19.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 210453397507' 2024-06-03T07:19:19.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:19:20.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397507 -lt 210453397507 2024-06-03T07:19:20.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:19:20.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:19:20.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:19:20.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T07:19:20.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:19:20.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:19:20.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:19:20.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:19:20.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:19:20.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:19:20.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:19:20.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T07:19:20.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:19:20.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:19:20.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:19:21.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T07:19:21.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:19:21.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:19:21.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-06-03T07:19:21.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-03T07:19:21.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-06-03T07:19:21.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-06-03T07:19:21.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T07:19:21.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-06-03T07:19:21.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-06-03T07:19:21.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-03T07:19:21.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-03T07:19:21.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:19:21.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:19:21.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:19:21.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:19:21.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T07:18:45.961785+0000 2024-06-03T07:19:21.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-03T07:19:22.126 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-06-03T07:19:22.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-03T07:18:45.961785+0000 2024-06-03T07:19:22.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T07:19:22.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T07:18:45.961785+0000 2024-06-03T07:19:22.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T07:19:22.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:19:22.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:19:22.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:19:22.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:19:22.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:19:22.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:19:22.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:19:22.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:18:45.961785+0000 '>' 2024-06-03T07:18:45.961785+0000 2024-06-03T07:19:22.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:19:23.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:19:23.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:19:23.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:19:23.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:19:23.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:19:23.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:19:23.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:19:23.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:19:22.309219+0000 '>' 2024-06-03T07:18:45.961785+0000 2024-06-03T07:19:23.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:19:23.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-06-03T07:19:23.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-06-03T07:19:23.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-06-03T07:19:23.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-06-03T07:19:23.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-06-03T07:19:23.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-06-03T07:19:23.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-06-03T07:19:23.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-06-03T07:19:23.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-06-03T07:19:23.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T07:19:23.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:19:23.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:19:23.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:19:23.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:19:23.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:19:23.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:19:23.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:19:23.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:19:23.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:19:23.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:19:23.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:19:23.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:19:23.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:19:23.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:19:23.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:19:23.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:19:23.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:19:23.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:19:23.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:19:23.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:19:23.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:19:23.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:19:23.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:19:23.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:19:23.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:19:23.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:19:23.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:19:23.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:19:23.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T07:19:23.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T07:19:23.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T07:19:23.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T07:19:23.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:19:23.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:19:23.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:19:23.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:19:23.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:19:23.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:19:23.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:19:23.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:19:23.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:19:23.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:19:23.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:19:23.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:19:23.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:19:23.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:19:23.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:19:23.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:19:23.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:19:23.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:19:23.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:19:23.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:19:23.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:19:23.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:19:23.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:19:23.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:19:23.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:19:23.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:19:23.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:19:23.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:19:23.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T07:19:23.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T07:19:23.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:19:23.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:19:23.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T07:19:23.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T07:19:23.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T07:19:23.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T07:19:23.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T07:19:23.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-06-03T07:19:23.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5759: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-06-03T07:19:23.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5760: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-06-03T07:19:23.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5761: TEST_periodic_scrub_replicated: local objname=POBJ 2024-06-03T07:19:23.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_periodic_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-06-03T07:19:23.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T07:19:23.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T07:19:23.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T07:19:23.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T07:19:23.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T07:19:23.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-06-03T07:19:24.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T07:19:24.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:19:24.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:19:24.358 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:19:24.358 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:19:24.358 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:19:24.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:19:24.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-03T07:19:24.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T07:19:24.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T07:19:24.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:19:24.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:19:24.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T07:19:24.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T07:19:24.563 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:19:24.563 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:19:24.563 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:19:24.563 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:19:24.564 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:19:24.564 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:19:24.564 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:19:24.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:19:24.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T07:19:24.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T07:19:24.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:19:24.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:19:24.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T07:19:24.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T07:19:24.646 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:19:24.646 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:19:24.647 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:19:24.647 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:19:24.647 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:19:24.648 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:19:24.648 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:19:24.648 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:19:24.648 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T07:19:24.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5764: TEST_periodic_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-06-03T07:19:24.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T07:19:24.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T07:19:24.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T07:19:24.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T07:19:24.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T07:19:24.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T07:19:25.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T07:19:25.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:19:25.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:19:25.544 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:19:25.544 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:19:25.544 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:19:25.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:19:25.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:19:25.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:19:25.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5765: TEST_periodic_scrub_replicated: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-06-03T07:19:25.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5766: TEST_periodic_scrub_replicated: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-06-03T07:19:25.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5767: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-03T07:19:25.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:19:25.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:19:25.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T07:19:25.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:19:25.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:19:25.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:19:25.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:19:25.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:19:25.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:19:25.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:19:25.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:19:25.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:19:25.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:19:25.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:19:25.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:19:25.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:19:25.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:19:25.580 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:19:25.580 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:19:25.580 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:19:25.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:19:25.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:19:25.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:19:25.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:19:25.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:19:25.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:19:25.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:19:25.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:19:25.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:19:25.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:19:25.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:19:25.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:19:25.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-06-03T07:19:25.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:19:25.581 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:19:25.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0c5808a6-db09-433b-a088-43bbfd4424dd 2024-06-03T07:19:25.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0c5808a6-db09-433b-a088-43bbfd4424dd' 2024-06-03T07:19:25.582 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 0c5808a6-db09-433b-a088-43bbfd4424dd 2024-06-03T07:19:25.582 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:19:25.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9bl1m90S8IxAAAbqUWq9z56ZLdtC2OqHwnQ== 2024-06-03T07:19:25.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9bl1m90S8IxAAAbqUWq9z56ZLdtC2OqHwnQ=="}' 2024-06-03T07:19:25.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0c5808a6-db09-433b-a088-43bbfd4424dd -i td/osd-scrub-repair/0/new.json 2024-06-03T07:19:25.858 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:19:25.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T07:19:25.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQB9bl1m90S8IxAAAbqUWq9z56ZLdtC2OqHwnQ== --osd-uuid 0c5808a6-db09-433b-a088-43bbfd4424dd 2024-06-03T07:19:25.900 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:25.901+0000 7f9989a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:25.908 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:25.909+0000 7f9989a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:25.910 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:25.910+0000 7f9989a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:25.910 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:25.911+0000 7f9989a0c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:19:25.910 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:25.911+0000 7f9989a0c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:19:25.910 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:25.911+0000 7f9989a0c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:19:26.001 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:26.001+0000 7f9989a0c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T07:19:31.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T07:19:31.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:19:31.758 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T07:19:31.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T07:19:31.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:19:32.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T07:19:32.192 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T07:19:32.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-03T07:19:32.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:19:32.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:19:32.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:19:32.222 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:32.221+0000 7fd5a69b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:32.224 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:32.224+0000 7fd5a69b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:32.225 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:32.225+0000 7fd5a69b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:32.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T07:19:32.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:19:32.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:19:32.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:19:32.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:19:32.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:32.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:19:32.488 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:19:32.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:32.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:32.740 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:32.740+0000 7fd5a69b8640 -1 Falling back to public interface 2024-06-03T07:19:32.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:33.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:33.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:33.781 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:19:33.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:19:33.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:33.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:34.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:35.083 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:35.082+0000 7fd5a69b8640 -1 osd.0 0 log_to_monitors true 2024-06-03T07:19:35.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:35.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:35.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:19:35.086 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:19:35.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:35.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:35.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:36.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:36.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:36.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:19:36.519 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:19:36.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:36.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:36.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:37.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:37.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:37.819 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:19:37.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:19:37.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:37.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:38.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:39.117 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:19:39.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:39.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:39.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:19:39.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:39.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:39.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:40.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:40.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:40.425 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:19:40.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:19:40.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:40.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:40.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:41.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:41.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:41.727 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:19:41.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:19:41.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:41.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:42.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:43.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:43.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:43.058 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:19:43.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:19:43.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:43.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:43.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:44.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:44.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:44.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:19:44.357 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:19:44.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:44.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:44.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:45.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:45.679 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:19:45.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:45.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:19:45.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:45.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:46.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:47.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:47.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:47.011 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:19:47.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:19:47.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:47.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:47.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:48.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:48.401 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:19:48.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:48.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:19:48.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:48.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:48.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:49.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:49.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:49.918 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:19:49.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:19:49.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:49.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:50.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:51.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:51.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:51.486 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:19:51.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:19:51.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:51.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:52.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:53.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:53.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:53.264 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:19:53.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:19:53.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:53.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:53.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:54.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:54.790 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:19:54.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:54.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:19:54.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:54.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:55.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:56.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:56.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:56.410 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:19:56.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:19:56.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:56.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:57.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:19:58.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:19:58.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:19:58.121 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T07:19:58.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T07:19:58.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:19:58.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:19:58.419 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3080143130,v1:127.0.0.1:6803/3080143130] [v2:127.0.0.1:6804/3080143130,v1:127.0.0.1:6805/3080143130] exists,up 0c5808a6-db09-433b-a088-43bbfd4424dd 2024-06-03T07:19:58.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:19:58.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:19:58.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:19:58.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5768: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-03T07:19:58.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:19:58.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:19:58.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T07:19:58.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:19:58.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:19:58.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:19:58.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:19:58.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:19:58.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:19:58.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:19:58.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:19:58.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:19:58.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:19:58.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:19:58.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:19:58.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:19:58.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:19:58.425 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:19:58.425 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:19:58.425 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:19:58.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:19:58.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:19:58.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:19:58.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:19:58.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:19:58.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:19:58.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:19:58.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:19:58.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:19:58.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:19:58.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:19:58.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:19:58.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-06-03T07:19:58.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:19:58.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:19:58.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d9e65576-9ade-4fbf-bcaf-9bd7ef71ca7d 2024-06-03T07:19:58.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d9e65576-9ade-4fbf-bcaf-9bd7ef71ca7d' 2024-06-03T07:19:58.428 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 d9e65576-9ade-4fbf-bcaf-9bd7ef71ca7d 2024-06-03T07:19:58.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:19:58.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCebl1mBulvGhAATWGyAEg+5jpem7ClCi3trQ== 2024-06-03T07:19:58.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCebl1mBulvGhAATWGyAEg+5jpem7ClCi3trQ=="}' 2024-06-03T07:19:58.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d9e65576-9ade-4fbf-bcaf-9bd7ef71ca7d -i td/osd-scrub-repair/1/new.json 2024-06-03T07:19:58.804 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:19:58.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T07:19:58.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQCebl1mBulvGhAATWGyAEg+5jpem7ClCi3trQ== --osd-uuid d9e65576-9ade-4fbf-bcaf-9bd7ef71ca7d 2024-06-03T07:19:58.848 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:58.849+0000 7f23f1461640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:58.889 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:58.889+0000 7f23f1461640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:58.890 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:58.891+0000 7f23f1461640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:19:58.890 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:58.891+0000 7f23f1461640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:19:58.890 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:58.891+0000 7f23f1461640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:19:58.890 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:58.891+0000 7f23f1461640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:19:58.964 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:19:58.964+0000 7f23f1461640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T07:20:04.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T07:20:04.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:20:04.948 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T07:20:04.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T07:20:04.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:20:05.330 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T07:20:05.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T07:20:05.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-03T07:20:05.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:20:05.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:20:05.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:20:05.360 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:20:05.361+0000 7f2984c19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:20:05.363 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:20:05.364+0000 7f2984c19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:20:05.364 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:20:05.365+0000 7f2984c19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:20:05.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T07:20:05.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:20:05.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:20:05.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:20:05.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:20:05.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:05.622 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:20:05.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:20:05.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:05.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:05.898 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:20:05.899+0000 7f2984c19640 -1 Falling back to public interface 2024-06-03T07:20:06.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:07.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:07.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:07.012 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:20:07.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:20:07.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:07.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:07.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:08.228 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:20:08.229+0000 7f2984c19640 -1 osd.1 0 log_to_monitors true 2024-06-03T07:20:08.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:08.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:08.316 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:20:08.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:20:08.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:08.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:08.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:09.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:09.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:09.713 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:20:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:20:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:11.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:11.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:11.113 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:20:11.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:20:11.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:11.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:11.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:12.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:12.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:12.464 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:20:12.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:20:12.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:12.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:12.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:13.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:13.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:13.767 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:20:13.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:20:13.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:13.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:14.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:15.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:15.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:15.166 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:20:15.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:20:15.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:15.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:15.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:16.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:16.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:16.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:20:16.465 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:20:16.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:16.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:16.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:17.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:17.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:17.767 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:20:17.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:20:17.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:17.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:18.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:19.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:19.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:19.142 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:20:19.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:20:19.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:19.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:19.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:20.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:20.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:20.448 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:20:20.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:20:20.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:20.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:20.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:21.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:21.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:21.888 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:20:21.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:20:21.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:21.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:22.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:23.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:23.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:23.628 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:20:23.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:20:23.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:23.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:24.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:25.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:25.139 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:20:25.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:20:25.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:25.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:25.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:26.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:26.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:26.694 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:20:26.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:20:26.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:26.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:27.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:28.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:28.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:28.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:20:28.384 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:20:28.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:28.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:29.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:30.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:30.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:30.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:20:30.023 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:20:30.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:30.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:30.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:31.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:31.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:31.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T07:20:31.316 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T07:20:31.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:31.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:20:31.636 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1090311509,v1:127.0.0.1:6811/1090311509] [v2:127.0.0.1:6812/1090311509,v1:127.0.0.1:6813/1090311509] exists,up d9e65576-9ade-4fbf-bcaf-9bd7ef71ca7d 2024-06-03T07:20:31.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:20:31.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:20:31.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:20:31.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5769: TEST_periodic_scrub_replicated: create_rbd_pool 2024-06-03T07:20:31.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T07:20:31.923 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-06-03T07:20:31.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T07:20:31.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T07:20:32.342 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' created 2024-06-03T07:20:32.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:20:33.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T07:20:35.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5770: TEST_periodic_scrub_replicated: wait_for_clean 2024-06-03T07:20:35.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:20:35.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:20:35.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:20:35.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:20:35.121 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:20:35.122 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:20:35.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:20:35.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:20:35.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:20:35.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:20:35.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:20:35.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:20:35.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:20:35.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:20:35.233 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:20:35.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:20:35.521 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:20:35.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:20:35.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:20:35.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:20:35.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836490 2024-06-03T07:20:35.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836490 2024-06-03T07:20:35.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490' 2024-06-03T07:20:35.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:20:35.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:20:35.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-06-03T07:20:35.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-06-03T07:20:35.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490 1-42949672964' 2024-06-03T07:20:35.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:20:35.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836490 2024-06-03T07:20:35.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:20:35.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:20:35.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836490 2024-06-03T07:20:35.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:20:35.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836490 2024-06-03T07:20:35.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-06-03T07:20:35.766 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836490 2024-06-03T07:20:35.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:20:36.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836490 2024-06-03T07:20:36.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:20:36.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-06-03T07:20:36.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:20:36.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:20:36.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-06-03T07:20:36.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:20:36.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-06-03T07:20:36.092 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672964 2024-06-03T07:20:36.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-06-03T07:20:36.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:20:36.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672963 -lt 42949672964 2024-06-03T07:20:36.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:20:37.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:20:37.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:20:37.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672963 -lt 42949672964 2024-06-03T07:20:37.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:20:38.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T07:20:38.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:20:39.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672964 2024-06-03T07:20:39.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:20:39.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:20:39.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:20:39.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T07:20:39.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:20:39.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:20:39.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:20:39.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:20:39.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:20:39.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:20:39.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:20:39.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T07:20:39.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:20:39.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:20:39.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:20:40.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T07:20:40.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:20:40.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:20:40.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5772: TEST_periodic_scrub_replicated: create_pool psr_pool 1 1 2024-06-03T07:20:40.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 1 1 2024-06-03T07:20:40.558 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'psr_pool' created 2024-06-03T07:20:40.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:20:41.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5773: TEST_periodic_scrub_replicated: wait_for_clean 2024-06-03T07:20:41.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:20:41.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:20:41.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:20:41.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:20:41.573 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:20:41.573 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:20:41.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:20:41.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:20:41.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:20:41.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:20:41.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:20:41.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:20:41.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:20:41.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:20:41.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:20:41.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:20:41.992 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:20:41.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:20:41.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:20:41.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:20:42.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-06-03T07:20:42.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-06-03T07:20:42.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-06-03T07:20:42.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:20:42.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:20:42.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-06-03T07:20:42.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-06-03T07:20:42.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-42949672967' 2024-06-03T07:20:42.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:20:42.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-06-03T07:20:42.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:20:42.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:20:42.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-06-03T07:20:42.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:20:42.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-06-03T07:20:42.239 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836492 2024-06-03T07:20:42.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-06-03T07:20:42.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:20:42.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-06-03T07:20:42.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:20:43.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:20:43.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:20:43.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836492 2024-06-03T07:20:43.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:20:43.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-06-03T07:20:43.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:20:43.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:20:43.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-06-03T07:20:43.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:20:43.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-06-03T07:20:43.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-06-03T07:20:43.832 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672967 2024-06-03T07:20:43.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:20:44.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-06-03T07:20:44.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:20:44.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:20:44.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:20:44.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T07:20:44.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:20:44.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:20:44.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:20:44.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:20:44.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:20:44.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:20:44.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:20:44.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T07:20:44.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:20:44.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:20:44.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:20:45.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T07:20:45.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:20:45.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:20:45.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5775: TEST_periodic_scrub_replicated: local osd=0 2024-06-03T07:20:45.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5776: TEST_periodic_scrub_replicated: add_something td/osd-scrub-repair psr_pool POBJ scrub 2024-06-03T07:20:45.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-03T07:20:45.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-06-03T07:20:45.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-06-03T07:20:45.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=scrub 2024-06-03T07:20:45.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' scrub = noscrub ']' 2024-06-03T07:20:45.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:81: add_something: ceph osd unset noscrub 2024-06-03T07:20:45.552 INFO:tasks.workunit.client.0.smithi039.stderr:noscrub is unset 2024-06-03T07:20:45.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:82: add_something: ceph osd unset nodeep-scrub 2024-06-03T07:20:45.963 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is unset 2024-06-03T07:20:45.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-03T07:20:45.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-03T07:20:45.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-06-03T07:20:46.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: get_primary psr_pool POBJ 2024-06-03T07:20:46.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-06-03T07:20:46.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-06-03T07:20:46.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-06-03T07:20:46.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T07:20:46.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: local primary=0 2024-06-03T07:20:46.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: get_pg psr_pool POBJ 2024-06-03T07:20:46.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-06-03T07:20:46.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-06-03T07:20:46.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-06-03T07:20:46.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T07:20:46.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: local pg=2.0 2024-06-03T07:20:46.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5781: TEST_periodic_scrub_replicated: local payload=UVWXYZ 2024-06-03T07:20:46.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5782: TEST_periodic_scrub_replicated: echo UVWXYZ 2024-06-03T07:20:46.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5784: TEST_periodic_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:20:46.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-03T07:20:46.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T07:20:46.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T07:20:46.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T07:20:46.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:20:46.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-03T07:20:46.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T07:20:46.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T07:20:46.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T07:20:46.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:20:46.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:20:46.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:20:46.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:20:46.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:20:46.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:20:47.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:20:47.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:20:47.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:20:47.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:20:47.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:20:47.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:20:47.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:20:47.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-06-03T07:20:49.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-03T07:20:49.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:20:49.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:20:49.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:20:49.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:20:49.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:20:49.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:20:49.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:20:49.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:20:49.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:20:49.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:20:49.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:20:49.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:20:49.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:20:49.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:20:49.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:20:49.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:20:49.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:20:49.103 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:20:49.103 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:20:49.103 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:20:49.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:20:49.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:20:49.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:20:49.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:20:49.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:20:49.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:20:49.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:20:49.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:20:49.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:20:49.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:20:49.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-06-03T07:20:49.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:20:49.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:20:49.105 INFO:tasks.workunit.client.0.smithi039.stderr:start osd.0 2024-06-03T07:20:49.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-03T07:20:49.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:20:49.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:20:49.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:20:49.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:20:49.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:20:49.135 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:20:49.136+0000 7fa49f21a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:20:49.138 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:20:49.138+0000 7fa49f21a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:20:49.139 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:20:49.140+0000 7fa49f21a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:20:49.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:20:49.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:20:49.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:20:49.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:20:49.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:20:49.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:49.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:20:49.405 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-06-03T07:20:49.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:49.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:20:49.654 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:20:49.655+0000 7fa49f21a640 -1 Falling back to public interface 2024-06-03T07:20:49.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:50.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:50.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:50.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:20:50.695 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:20:50.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:50.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:20:50.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:51.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:51.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:51.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:20:51.996 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-06-03T07:20:51.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:51.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:20:52.108 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:20:52.108+0000 7fa49f21a640 -1 osd.0 25 log_to_monitors true 2024-06-03T07:20:52.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:20:53.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:20:53.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:20:53.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:20:53.312 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-06-03T07:20:53.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:20:53.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:20:53.786 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [5,25) [v2:127.0.0.1:6802/1249596594,v1:127.0.0.1:6803/1249596594] [v2:127.0.0.1:6804/1249596594,v1:127.0.0.1:6805/1249596594] exists,up 0c5808a6-db09-433b-a088-43bbfd4424dd 2024-06-03T07:20:53.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:20:53.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:20:53.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:20:53.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T07:20:53.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:20:53.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:20:53.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:20:53.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:20:53.788 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:20:53.788 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:20:53.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:20:53.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:20:53.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:20:53.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:20:53.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:20:53.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:20:53.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:20:53.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:20:53.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:20:54.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:20:54.195 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:20:54.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:20:54.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:20:54.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:20:54.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-06-03T07:20:54.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-06-03T07:20:54.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-06-03T07:20:54.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:20:54.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:20:54.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-06-03T07:20:54.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-06-03T07:20:54.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672970' 2024-06-03T07:20:54.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:20:54.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-06-03T07:20:54.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:20:54.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:20:54.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-06-03T07:20:54.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:20:54.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-06-03T07:20:54.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-06-03T07:20:54.416 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.0 seq 124554051586 2024-06-03T07:20:54.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:20:54.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-06-03T07:20:54.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:20:55.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:20:55.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:20:56.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-06-03T07:20:56.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:20:56.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-06-03T07:20:56.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:20:56.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:20:56.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-06-03T07:20:56.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:20:56.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-06-03T07:20:56.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-06-03T07:20:56.096 INFO:tasks.workunit.client.0.smithi039.stderr:waiting osd.1 seq 42949672970 2024-06-03T07:20:56.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:20:56.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-06-03T07:20:56.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:20:56.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:20:56.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:20:56.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-03T07:20:56.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:20:56.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:20:56.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:20:56.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:20:56.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:20:56.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:20:56.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:20:57.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-03T07:20:57.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:20:57.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:20:57.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:20:57.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-03T07:20:57.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:20:57.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:20:57.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5787: TEST_periodic_scrub_replicated: set -o pipefail 2024-06-03T07:20:57.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-06-03T07:20:57.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: jq . 2024-06-03T07:20:57.425 INFO:tasks.workunit.client.0.smithi039.stderr:No scrub information available for pg 2.0 2024-06-03T07:20:57.425 INFO:tasks.workunit.client.0.smithi039.stderr:error 2: (2) No such file or directory 2024-06-03T07:20:57.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5789: TEST_periodic_scrub_replicated: set +o pipefail 2024-06-03T07:20:57.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5791: TEST_periodic_scrub_replicated: pg_deep_scrub 2.0 2024-06-03T07:20:57.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=2.0 2024-06-03T07:20:57.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-03T07:20:57.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:20:57.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:20:57.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:20:57.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-03T07:20:57.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-06-03T07:20:40.424893+0000 2024-06-03T07:20:57.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 2.0 2024-06-03T07:20:58.024 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0 on osd.0 to deep-scrub 2024-06-03T07:20:58.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 2.0 2024-06-03T07:20:40.424893+0000 last_deep_scrub_stamp 2024-06-03T07:20:58.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T07:20:58.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T07:20:40.424893+0000 2024-06-03T07:20:58.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-06-03T07:20:58.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:20:58.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:20:58.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-03T07:20:58.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:20:58.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:20:58.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:20:58.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-03T07:20:58.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:20:40.424893+0000 '>' 2024-06-03T07:20:40.424893+0000 2024-06-03T07:20:58.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:20:59.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:20:59.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:20:59.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-03T07:20:59.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:20:59.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:20:59.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:20:59.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-03T07:20:59.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:20:40.424893+0000 '>' 2024-06-03T07:20:40.424893+0000 2024-06-03T07:20:59.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:21:00.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:21:00.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:21:00.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-03T07:21:00.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:21:00.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:21:00.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:21:00.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-03T07:21:00.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:20:40.424893+0000 '>' 2024-06-03T07:20:40.424893+0000 2024-06-03T07:21:00.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:21:01.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:21:01.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:21:01.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-03T07:21:01.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:21:01.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:21:01.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:21:01.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-03T07:21:02.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:20:40.424893+0000 '>' 2024-06-03T07:20:40.424893+0000 2024-06-03T07:21:02.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:21:03.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:21:03.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:21:03.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-03T07:21:03.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:21:03.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:21:03.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:21:03.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-03T07:21:03.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:20:40.424893+0000 '>' 2024-06-03T07:20:40.424893+0000 2024-06-03T07:21:03.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:21:04.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:21:04.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:21:04.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-03T07:21:04.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:21:04.552 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-03T07:21:04.552 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:21:04.552 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-03T07:21:04.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:20:58.197312+0000 '>' 2024-06-03T07:20:40.424893+0000 2024-06-03T07:21:04.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:21:04.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-06-03T07:21:04.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: jq . 2024-06-03T07:21:04.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: grep -q POBJ 2024-06-03T07:21:04.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5796: TEST_periodic_scrub_replicated: flush_pg_stats 2024-06-03T07:21:04.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:21:04.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:21:05.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:21:05.189 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:21:05.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:21:05.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:21:05.189 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:21:05.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-06-03T07:21:05.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-06-03T07:21:05.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589' 2024-06-03T07:21:05.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:21:05.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:21:05.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-06-03T07:21:05.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-06-03T07:21:05.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-42949672973' 2024-06-03T07:21:05.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:21:05.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051589 2024-06-03T07:21:05.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:21:05.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:21:05.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051589 2024-06-03T07:21:05.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:21:05.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-06-03T07:21:05.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-06-03T07:21:05.410 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 124554051589 2024-06-03T07:21:05.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:21:05.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051589 2024-06-03T07:21:05.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:21:06.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:21:06.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:21:07.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051589 2024-06-03T07:21:07.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:21:07.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-06-03T07:21:07.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:21:07.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:21:07.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-06-03T07:21:07.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:21:07.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-06-03T07:21:07.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-06-03T07:21:07.007 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672973 2024-06-03T07:21:07.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:21:07.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-06-03T07:21:07.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: get_last_scrub_stamp 2.0 2024-06-03T07:21:07.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:21:07.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:21:07.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:21:07.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:21:07.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: local last_scrub=2024-06-03T07:20:58.197312+0000 2024-06-03T07:21:07.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5799: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-06-03T07:21:07.730 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T07:21:07.730 INFO:tasks.workunit.client.0.smithi039.stdout: "deep": false, 2024-06-03T07:21:07.731 INFO:tasks.workunit.client.0.smithi039.stdout: "must": false, 2024-06-03T07:21:07.731 INFO:tasks.workunit.client.0.smithi039.stdout: "stamp": "2024-05-27T07:19:27.731898+0000" 2024-06-03T07:21:07.731 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T07:21:07.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5801: TEST_periodic_scrub_replicated: wait_for_scrub 2.0 2024-06-03T07:20:58.197312+0000 2024-06-03T07:21:07.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T07:21:07.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T07:20:58.197312+0000 2024-06-03T07:21:07.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T07:21:07.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:21:07.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:21:07.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:21:07.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:21:07.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:21:07.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:21:07.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:21:08.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:20:58.197312+0000 '>' 2024-06-03T07:20:58.197312+0000 2024-06-03T07:21:08.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:21:09.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:21:09.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:21:09.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:21:09.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:21:09.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:21:09.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:21:09.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:21:09.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:20:58.197312+0000 '>' 2024-06-03T07:20:58.197312+0000 2024-06-03T07:21:09.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:21:10.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:21:10.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:21:10.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:21:10.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:21:10.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:21:10.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:21:10.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:21:10.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:20:58.197312+0000 '>' 2024-06-03T07:20:58.197312+0000 2024-06-03T07:21:10.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:21:11.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:21:11.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:21:11.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:21:11.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:21:11.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:21:11.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:21:11.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:21:11.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:20:58.197312+0000 '>' 2024-06-03T07:20:58.197312+0000 2024-06-03T07:21:11.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:21:12.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:21:12.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:21:12.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:21:12.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:21:12.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:21:12.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:21:12.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:21:13.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:20:58.197312+0000 '>' 2024-06-03T07:20:58.197312+0000 2024-06-03T07:21:13.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:21:14.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:21:14.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:21:14.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:21:14.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:21:14.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:21:14.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:21:14.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:21:14.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:21:08.128918+0000 '>' 2024-06-03T07:20:58.197312+0000 2024-06-03T07:21:14.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:21:14.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5804: TEST_periodic_scrub_replicated: grep -q 'Deep scrub errors, upgrading scrub to deep-scrub' td/osd-scrub-repair/osd.0.log 2024-06-03T07:21:14.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-06-03T07:21:14.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: jq . 2024-06-03T07:21:14.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: grep -q POBJ 2024-06-03T07:21:14.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5810: TEST_periodic_scrub_replicated: ceph osd set nodeep-scrub 2024-06-03T07:21:14.994 INFO:tasks.workunit.client.0.smithi039.stderr:nodeep-scrub is set 2024-06-03T07:21:15.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5812: TEST_periodic_scrub_replicated: ceph tell osd.0 get_latest_osdmap 2024-06-03T07:21:15.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5813: TEST_periodic_scrub_replicated: flush_pg_stats 2024-06-03T07:21:15.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:21:15.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:21:15.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:21:15.407 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:21:15.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:21:15.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:21:15.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:21:15.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051592 2024-06-03T07:21:15.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051592 2024-06-03T07:21:15.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051592' 2024-06-03T07:21:15.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:21:15.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:21:15.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-06-03T07:21:15.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-06-03T07:21:15.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051592 1-42949672976' 2024-06-03T07:21:15.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:21:15.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051592 2024-06-03T07:21:15.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:21:15.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:21:15.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051592 2024-06-03T07:21:15.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:21:15.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051592 2024-06-03T07:21:15.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051592' 2024-06-03T07:21:15.679 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 124554051592 2024-06-03T07:21:15.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:21:15.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051592 -lt 124554051592 2024-06-03T07:21:15.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:21:15.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-06-03T07:21:15.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:21:15.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:21:15.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-06-03T07:21:15.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:21:15.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-06-03T07:21:15.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-06-03T07:21:15.918 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672976 2024-06-03T07:21:15.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:21:16.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-06-03T07:21:16.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5814: TEST_periodic_scrub_replicated: sleep 5 2024-06-03T07:21:21.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5817: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-06-03T07:21:21.363 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T07:21:21.364 INFO:tasks.workunit.client.0.smithi039.stdout: "deep": false, 2024-06-03T07:21:21.364 INFO:tasks.workunit.client.0.smithi039.stdout: "must": false, 2024-06-03T07:21:21.364 INFO:tasks.workunit.client.0.smithi039.stdout: "stamp": "2024-05-27T07:19:41.365198+0000" 2024-06-03T07:21:21.364 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T07:21:21.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5820: TEST_periodic_scrub_replicated: local found=false 2024-06-03T07:21:21.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: seq 14 -1 0 2024-06-03T07:21:21.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: for i in $(seq 14 -1 0) 2024-06-03T07:21:21.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5823: TEST_periodic_scrub_replicated: sleep 1 2024-06-03T07:21:22.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: grep -q 'Regular scrub skipped due to deep-scrub errors and nodeep-scrub set' td/osd-scrub-repair/osd.0.log 2024-06-03T07:21:22.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: found=true 2024-06-03T07:21:22.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: break 2024-06-03T07:21:22.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5827: TEST_periodic_scrub_replicated: test true = true 2024-06-03T07:21:22.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-06-03T07:21:22.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: jq . 2024-06-03T07:21:22.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: grep -q POBJ 2024-06-03T07:21:22.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5832: TEST_periodic_scrub_replicated: flush_pg_stats 2024-06-03T07:21:22.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:21:22.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:21:22.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:21:22.746 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:21:22.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:21:22.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:21:22.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:21:22.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051595 2024-06-03T07:21:22.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051595 2024-06-03T07:21:22.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051595' 2024-06-03T07:21:22.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:21:22.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:21:22.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-06-03T07:21:22.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-06-03T07:21:22.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051595 1-42949672979' 2024-06-03T07:21:22.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:21:22.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051595 2024-06-03T07:21:22.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:21:22.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:21:22.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051595 2024-06-03T07:21:22.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:21:22.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051595 2024-06-03T07:21:22.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051595' 2024-06-03T07:21:22.967 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 124554051595 2024-06-03T07:21:22.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:21:23.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051593 -lt 124554051595 2024-06-03T07:21:23.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:21:24.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:21:24.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:21:24.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051595 -lt 124554051595 2024-06-03T07:21:24.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:21:24.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-06-03T07:21:24.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:21:24.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:21:24.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-06-03T07:21:24.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:21:24.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-06-03T07:21:24.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-06-03T07:21:24.578 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672979 2024-06-03T07:21:24.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:21:24.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-06-03T07:21:24.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5834: TEST_periodic_scrub_replicated: pg_schedule_scrub 2.0 2024-06-03T07:21:24.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: pg_schedule_scrub: local pgid=2.0 2024-06-03T07:21:24.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: get_last_scrub_stamp 2.0 2024-06-03T07:21:24.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:21:24.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:21:24.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:21:24.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:21:25.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: local last_scrub=2024-05-27T07:19:41.365198+0000 2024-06-03T07:21:25.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: pg_schedule_scrub: ceph pg scrub 2.0 2024-06-03T07:21:25.432 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 2.0 on osd.0 to scrub 2024-06-03T07:21:25.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: pg_schedule_scrub: wait_for_scrub 2.0 2024-05-27T07:19:41.365198+0000 2024-06-03T07:21:25.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-03T07:21:25.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-05-27T07:19:41.365198+0000 2024-06-03T07:21:25.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T07:21:25.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:21:25.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:21:25.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:21:25.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:21:25.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:21:25.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:21:25.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:21:25.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-05-27T07:19:41.365198+0000 '>' 2024-05-27T07:19:41.365198+0000 2024-06-03T07:21:25.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:21:26.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:21:26.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:21:26.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:21:26.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:21:26.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:21:26.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:21:26.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:21:27.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-05-27T07:19:41.365198+0000 '>' 2024-05-27T07:19:41.365198+0000 2024-06-03T07:21:27.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:21:28.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:21:28.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:21:28.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-03T07:21:28.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-03T07:21:28.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:21:28.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:21:28.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-03T07:21:28.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:21:26.191861+0000 '>' 2024-05-27T07:19:41.365198+0000 2024-06-03T07:21:28.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:21:28.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5835: TEST_periodic_scrub_replicated: grep -q 'Regular scrub request, deep-scrub details will be lost' td/osd-scrub-repair/osd.0.log 2024-06-03T07:21:28.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-06-03T07:21:28.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: jq . 2024-06-03T07:21:28.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: grep -qv POBJ 2024-06-03T07:21:28.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T07:21:28.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:21:28.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:21:28.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:21:28.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:21:28.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:21:28.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:21:28.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:21:28.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:21:28.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:21:28.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:21:28.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:21:28.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:21:28.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:21:28.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:21:28.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:21:28.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:21:28.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:21:28.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:21:28.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:21:28.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:21:28.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:21:28.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:21:28.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:21:28.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:21:28.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:21:28.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:21:28.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:21:28.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:21:28.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T07:21:28.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T07:21:28.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T07:21:28.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T07:21:28.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:21:28.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:21:28.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:21:28.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:21:28.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:21:28.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:21:28.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:21:28.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:21:28.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:21:28.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:21:28.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:21:28.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:21:28.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:21:28.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:21:28.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:21:28.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:21:28.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:21:28.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:21:28.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:21:28.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:21:28.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:21:28.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:21:28.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:21:28.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:21:28.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:21:28.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:21:28.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:21:28.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:21:28.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T07:21:28.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T07:21:28.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:21:28.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:21:28.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T07:21:28.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T07:21:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T07:21:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T07:21:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T07:21:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2024-06-03T07:21:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-06-03T07:21:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local poolname=testpool 2024-06-03T07:21:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local OSDS=2 2024-06-03T07:21:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OBJS=30 2024-06-03T07:21:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: local REPAIRS=20 2024-06-03T07:21:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-06-03T07:21:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T07:21:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T07:21:28.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T07:21:28.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T07:21:28.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T07:21:28.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T07:21:28.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T07:21:28.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:21:28.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:21:28.881 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:21:28.881 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:21:28.881 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:21:28.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:21:28.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T07:21:29.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T07:21:29.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T07:21:29.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:21:29.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:21:29.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T07:21:29.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T07:21:29.067 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:21:29.067 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:21:29.067 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:21:29.067 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:21:29.068 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:21:29.068 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:21:29.068 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:21:29.068 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:21:29.068 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T07:21:29.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T07:21:29.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:21:29.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:21:29.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T07:21:29.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T07:21:29.152 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:21:29.152 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:21:29.152 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:21:29.153 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:21:29.153 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:21:29.153 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:21:29.153 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:21:29.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:21:29.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T07:21:29.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-06-03T07:21:29.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T07:21:29.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T07:21:29.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T07:21:29.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T07:21:29.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T07:21:29.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T07:21:30.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T07:21:30.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:21:30.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:21:30.051 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:21:30.051 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:21:30.051 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:21:30.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:21:30.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:21:30.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:21:30.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:665: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T07:21:30.081 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: expr 2 - 1 2024-06-03T07:21:30.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: seq 0 1 2024-06-03T07:21:30.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T07:21:30.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:21:30.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:21:30.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:21:30.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T07:21:30.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:21:30.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:21:30.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:21:30.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:21:30.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:21:30.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:21:30.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:21:30.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:21:30.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:21:30.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:21:30.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:21:30.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:21:30.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:21:30.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:21:30.088 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:21:30.088 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:21:30.088 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:21:30.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:21:30.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:21:30.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:21:30.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:21:30.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:21:30.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:21:30.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:21:30.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:21:30.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:21:30.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:21:30.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:21:30.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:21:30.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T07:21:30.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:21:30.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:21:30.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cb8d7329-3611-489e-aea4-2ac5bcc34512 2024-06-03T07:21:30.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cb8d7329-3611-489e-aea4-2ac5bcc34512' 2024-06-03T07:21:30.090 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 cb8d7329-3611-489e-aea4-2ac5bcc34512 2024-06-03T07:21:30.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:21:30.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD6bl1mDYhoBhAAZgN2h07ORAS2mdIoGNckxA== 2024-06-03T07:21:30.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD6bl1mDYhoBhAAZgN2h07ORAS2mdIoGNckxA=="}' 2024-06-03T07:21:30.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cb8d7329-3611-489e-aea4-2ac5bcc34512 -i td/osd-scrub-repair/0/new.json 2024-06-03T07:21:30.356 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:21:30.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T07:21:30.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD6bl1mDYhoBhAAZgN2h07ORAS2mdIoGNckxA== --osd-uuid cb8d7329-3611-489e-aea4-2ac5bcc34512 2024-06-03T07:21:30.398 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:21:30.399+0000 7fa0a0811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:21:30.424 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:21:30.424+0000 7fa0a0811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:21:30.426 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:21:30.426+0000 7fa0a0811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:21:30.426 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:21:30.426+0000 7fa0a0811640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:21:30.426 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:21:30.426+0000 7fa0a0811640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:21:30.427 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:21:30.426+0000 7fa0a0811640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:21:30.516 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:21:30.516+0000 7fa0a0811640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T07:21:36.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T07:21:36.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:21:36.270 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T07:21:36.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T07:21:36.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:21:36.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T07:21:36.653 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T07:21:36.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:21:36.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:21:36.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:21:36.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:21:36.682 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:21:36.682+0000 7f5d5760f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:21:36.684 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:21:36.685+0000 7f5d5760f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:21:36.685 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:21:36.686+0000 7f5d5760f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:21:36.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T07:21:36.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:21:36.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:21:36.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:21:36.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:21:36.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:36.944 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:21:36.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:21:36.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:36.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:37.199 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:21:37.200+0000 7f5d5760f640 -1 Falling back to public interface 2024-06-03T07:21:37.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:38.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:38.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:38.233 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:21:38.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:21:38.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:38.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:38.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:39.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:39.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:39.538 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:21:39.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:21:39.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:39.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:39.584 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:21:39.584+0000 7f5d5760f640 -1 osd.0 0 log_to_monitors true 2024-06-03T07:21:39.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:40.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:40.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:40.844 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:21:40.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:21:40.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:40.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:41.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:42.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:42.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:42.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:21:42.182 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:21:42.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:42.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:42.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:43.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:43.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:43.537 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:21:43.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:21:43.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:43.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:43.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:44.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:44.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:44.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:21:44.843 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:21:44.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:44.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:45.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:46.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:46.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:46.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:21:46.148 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:21:46.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:46.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:46.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:47.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:47.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:47.568 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:21:47.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:21:47.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:47.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:47.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:48.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:48.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:48.867 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:21:48.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:21:48.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:48.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:49.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:50.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:50.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:50.170 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:21:50.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:21:50.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:50.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:50.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:51.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:51.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:51.597 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:21:51.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:21:51.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:51.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:51.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:52.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:52.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:52.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:21:52.981 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:21:52.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:52.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:53.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:54.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:54.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:54.555 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:21:54.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:21:54.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:54.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:55.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:56.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:56.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:56.140 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:21:56.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:21:56.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:56.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:56.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:57.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:57.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:57.850 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:21:57.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:21:57.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:57.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:58.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:21:59.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:21:59.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:21:59.337 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:21:59.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:21:59.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:21:59.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:21:59.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:00.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:00.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:00.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:22:00.915 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:22:00.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:00.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:22:01.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:02.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:02.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:02.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T07:22:02.365 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T07:22:02.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:02.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:22:02.669 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2616245079,v1:127.0.0.1:6803/2616245079] [v2:127.0.0.1:6804/2616245079,v1:127.0.0.1:6805/2616245079] exists,up cb8d7329-3611-489e-aea4-2ac5bcc34512 2024-06-03T07:22:02.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:22:02.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:22:02.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:22:02.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T07:22:02.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:22:02.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:22:02.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:22:02.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T07:22:02.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:22:02.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:22:02.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:22:02.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:22:02.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:22:02.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:22:02.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:22:02.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:22:02.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:22:02.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:22:02.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:22:02.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:22:02.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:22:02.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:22:02.675 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:22:02.675 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:22:02.675 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:22:02.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:22:02.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:22:02.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:22:02.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:22:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:22:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:22:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:22:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:22:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:22:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:22:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:22:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:22:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T07:22:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:22:02.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:22:02.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=232342c9-70d4-4707-b07d-894e2f508ca2 2024-06-03T07:22:02.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 232342c9-70d4-4707-b07d-894e2f508ca2' 2024-06-03T07:22:02.677 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 232342c9-70d4-4707-b07d-894e2f508ca2 2024-06-03T07:22:02.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:22:02.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAab11mKzdTKRAAWAcfpzY3tISD8HtDr4k6Dg== 2024-06-03T07:22:02.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAab11mKzdTKRAAWAcfpzY3tISD8HtDr4k6Dg=="}' 2024-06-03T07:22:02.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 232342c9-70d4-4707-b07d-894e2f508ca2 -i td/osd-scrub-repair/1/new.json 2024-06-03T07:22:03.159 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:22:03.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T07:22:03.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAab11mKzdTKRAAWAcfpzY3tISD8HtDr4k6Dg== --osd-uuid 232342c9-70d4-4707-b07d-894e2f508ca2 2024-06-03T07:22:03.202 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:22:03.203+0000 7f2989e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:22:03.235 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:22:03.236+0000 7f2989e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:22:03.236 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:22:03.237+0000 7f2989e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:22:03.237 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:22:03.237+0000 7f2989e11640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:22:03.237 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:22:03.237+0000 7f2989e11640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:22:03.237 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:22:03.237+0000 7f2989e11640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:22:03.311 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:22:03.312+0000 7f2989e11640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T07:22:09.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T07:22:09.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:22:09.486 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T07:22:09.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T07:22:09.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:22:09.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T07:22:09.866 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T07:22:09.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:22:09.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:22:09.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:22:09.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:22:09.894 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:22:09.894+0000 7f893616e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:22:09.896 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:22:09.896+0000 7f893616e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:22:09.898 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:22:09.897+0000 7f893616e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:22:10.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T07:22:10.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:22:10.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:22:10.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:22:10.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:22:10.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:10.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:22:10.154 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:22:10.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:10.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:10.425 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:22:10.425+0000 7f893616e640 -1 Falling back to public interface 2024-06-03T07:22:10.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:11.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:11.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:11.446 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:22:11.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:22:11.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:11.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:11.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:12.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:12.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:12.746 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:22:12.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:22:12.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:12.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:12.837 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:22:12.836+0000 7f893616e640 -1 osd.1 0 log_to_monitors true 2024-06-03T07:22:13.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:14.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:14.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:14.050 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:22:14.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:22:14.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:14.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:14.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:15.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:15.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:15.370 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:22:15.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:22:15.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:15.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:15.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:16.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:16.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:16.715 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:22:16.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:22:16.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:16.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:17.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:18.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:18.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:18.020 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:22:18.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:22:18.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:18.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:18.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:19.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:19.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:19.345 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:22:19.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:22:19.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:19.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:19.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:20.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:20.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:20.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:22:20.681 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:22:20.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:20.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:20.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:21.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:21.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:21.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:22:21.983 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:22:21.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:21.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:22.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:23.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:23.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:23.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:22:23.284 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:22:23.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:23.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:23.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:24.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:24.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:24.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:22:24.631 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:22:24.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:24.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:24.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:25.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:25.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:25.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:22:25.934 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:22:25.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:25.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:26.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:27.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:27.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:27.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:22:27.492 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:22:27.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:27.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:27.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:28.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:28.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:28.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:22:28.816 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:22:28.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:28.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:29.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:30.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:30.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:30.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:22:30.253 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:22:30.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:30.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:30.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:31.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:31.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:31.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:22:31.680 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:22:31.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:31.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:32.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:33.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:33.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:33.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:22:33.274 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:22:33.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:33.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:33.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:34.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:34.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:34.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T07:22:34.855 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T07:22:34.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:34.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:35.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:22:36.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:22:36.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:22:36.345 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-06-03T07:22:36.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T07:22:36.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:22:36.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:22:36.648 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1867562133,v1:127.0.0.1:6811/1867562133] [v2:127.0.0.1:6812/1867562133,v1:127.0.0.1:6813/1867562133] exists,up 232342c9-70d4-4707-b07d-894e2f508ca2 2024-06-03T07:22:36.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:22:36.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:22:36.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:22:36.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:670: TEST_repair_stats: create_pool testpool 1 1 2024-06-03T07:22:36.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-06-03T07:22:37.181 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'testpool' created 2024-06-03T07:22:37.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:22:38.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: ceph osd pool set testpool size 2 2024-06-03T07:22:38.653 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 2 2024-06-03T07:22:38.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: wait_for_clean 2024-06-03T07:22:38.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:22:38.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:22:38.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:22:38.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:22:38.666 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:22:38.666 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:22:38.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:22:38.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:22:38.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:22:38.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:22:38.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:22:38.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:22:38.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:22:38.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:22:38.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:22:39.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:22:39.091 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:22:39.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:22:39.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:22:39.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:22:39.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-06-03T07:22:39.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-06-03T07:22:39.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-06-03T07:22:39.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:22:39.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:22:39.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-06-03T07:22:39.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-06-03T07:22:39.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672966' 2024-06-03T07:22:39.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:22:39.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-06-03T07:22:39.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:22:39.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:22:39.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-06-03T07:22:39.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:22:39.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-06-03T07:22:39.426 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836496 2024-06-03T07:22:39.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-06-03T07:22:39.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:22:39.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836496 2024-06-03T07:22:39.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:22:40.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:22:40.821 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:22:41.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-06-03T07:22:41.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:22:41.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-06-03T07:22:41.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:22:41.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:22:41.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-06-03T07:22:41.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:22:41.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-06-03T07:22:41.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-06-03T07:22:41.123 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672966 2024-06-03T07:22:41.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:22:41.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672966 2024-06-03T07:22:41.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:22:41.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:22:41.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:22:41.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T07:22:41.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:22:41.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:22:41.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:22:41.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:22:41.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:22:41.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:22:41.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:22:42.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T07:22:42.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:22:42.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:22:42.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:22:42.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T07:22:42.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:22:42.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:22:42.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:675: TEST_repair_stats: local payload=ABCDEF 2024-06-03T07:22:42.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: echo ABCDEF 2024-06-03T07:22:42.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: seq 1 30 2024-06-03T07:22:42.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:42.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:42.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:42.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:42.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:42.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:42.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:42.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:43.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:43.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:43.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:43.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:43.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:43.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:43.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:43.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:43.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:43.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:43.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:43.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:43.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:43.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:43.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:43.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:44.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:44.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:44.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:44.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:44.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:44.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:44.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:44.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:44.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:44.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:44.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:44.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:44.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:44.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:45.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:45.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:45.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:45.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:45.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:45.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:45.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:45.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:45.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:45.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:45.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:45.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:45.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:45.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:45.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:45.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:46.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:46.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:46.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:46.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:46.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-03T07:22:46.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:22:46.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: get_not_primary testpool obj1 2024-06-03T07:22:46.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-03T07:22:46.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-06-03T07:22:46.484 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-06-03T07:22:46.484 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T07:22:46.485 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-03T07:22:46.485 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-03T07:22:46.485 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T07:22:46.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T07:22:46.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-06-03T07:22:46.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T07:22:47.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: local other=0 2024-06-03T07:22:47.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_pg testpool obj1 2024-06-03T07:22:47.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-03T07:22:47.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-06-03T07:22:47.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-06-03T07:22:47.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T07:22:47.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local pgid=1.0 2024-06-03T07:22:47.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_primary testpool obj1 2024-06-03T07:22:47.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T07:22:47.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-03T07:22:47.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-03T07:22:47.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T07:22:47.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local primary=1 2024-06-03T07:22:47.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:688: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:22:47.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:22:47.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:22:47.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:22:47.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:22:47.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:22:47.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:22:47.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:22:47.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:22:47.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:22:47.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:22:47.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:22:47.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:22:48.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:22:48.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: seq 1 20 2024-06-03T07:22:48.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:22:48.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 1 % 2 2024-06-03T07:22:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-03T07:22:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-06-03T07:22:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:22:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:22:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:22:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:22:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:22:48.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-06-03T07:22:49.438 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:ff7b1f36:::obj1:head# 2024-06-03T07:22:49.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:22:49.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 2 % 2 2024-06-03T07:22:49.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-03T07:22:49.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-06-03T07:22:49.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:22:49.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:22:49.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:22:49.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:22:49.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:22:49.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-06-03T07:22:51.098 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:104778fc:::obj2:head# 2024-06-03T07:22:51.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:22:51.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 3 % 2 2024-06-03T07:22:51.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-03T07:22:51.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-06-03T07:22:51.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:22:51.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:22:51.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:22:51.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:22:51.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:22:51.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-06-03T07:22:52.446 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:8dd16f86:::obj3:head# 2024-06-03T07:22:52.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:22:52.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 4 % 2 2024-06-03T07:22:52.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-03T07:22:52.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-06-03T07:22:52.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:22:52.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:22:52.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:22:52.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:22:52.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:22:52.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-06-03T07:22:53.774 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:0ee9ae15:::obj4:head# 2024-06-03T07:22:54.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:22:54.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 5 % 2 2024-06-03T07:22:54.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-03T07:22:54.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-06-03T07:22:54.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:22:54.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:22:54.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:22:54.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:22:54.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:22:54.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-06-03T07:22:54.944 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:c52c9666:::obj5:head# 2024-06-03T07:22:55.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:22:55.459 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 6 % 2 2024-06-03T07:22:55.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-03T07:22:55.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-06-03T07:22:55.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:22:55.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:22:55.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:22:55.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:22:55.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:22:55.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-06-03T07:22:56.147 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:233a42c1:::obj6:head# 2024-06-03T07:22:56.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:22:56.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 7 % 2 2024-06-03T07:22:56.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-03T07:22:56.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-06-03T07:22:56.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:22:56.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:22:56.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:22:56.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:22:56.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:22:56.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-06-03T07:22:57.416 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:9a09113a:::obj7:head# 2024-06-03T07:22:57.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:22:57.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 8 % 2 2024-06-03T07:22:57.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-03T07:22:57.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-06-03T07:22:57.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:22:57.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:22:57.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:22:57.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:22:57.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:22:57.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-06-03T07:22:58.651 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:a557efb1:::obj8:head# 2024-06-03T07:22:59.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:22:59.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 9 % 2 2024-06-03T07:22:59.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-03T07:22:59.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-06-03T07:22:59.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:22:59.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:22:59.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:22:59.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:22:59.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:22:59.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-06-03T07:22:59.821 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:fe351e27:::obj9:head# 2024-06-03T07:23:00.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:23:00.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 10 % 2 2024-06-03T07:23:00.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-03T07:23:00.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-06-03T07:23:00.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:23:00.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:23:00.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:23:00.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:23:00.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:23:00.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-06-03T07:23:01.039 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:d1337354:::obj10:head# 2024-06-03T07:23:01.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:23:01.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 11 % 2 2024-06-03T07:23:01.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-03T07:23:01.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-06-03T07:23:01.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:23:01.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:23:01.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:23:01.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:23:01.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:23:01.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-06-03T07:23:02.233 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:eeae2a94:::obj11:head# 2024-06-03T07:23:02.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:23:02.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 12 % 2 2024-06-03T07:23:03.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-03T07:23:03.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-06-03T07:23:03.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:23:03.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:23:03.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:23:03.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:23:03.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:23:03.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-06-03T07:23:03.678 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:af5f95cb:::obj12:head# 2024-06-03T07:23:04.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:23:04.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 13 % 2 2024-06-03T07:23:04.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-03T07:23:04.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-06-03T07:23:04.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:23:04.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:23:04.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:23:04.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:23:04.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:23:04.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-06-03T07:23:05.049 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:a61ad63e:::obj13:head# 2024-06-03T07:23:05.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:23:05.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 14 % 2 2024-06-03T07:23:05.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-03T07:23:05.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-06-03T07:23:05.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:23:05.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:23:05.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:23:05.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:23:05.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:23:05.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-06-03T07:23:06.184 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:e0a44829:::obj14:head# 2024-06-03T07:23:06.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:23:06.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 15 % 2 2024-06-03T07:23:06.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-03T07:23:06.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-06-03T07:23:06.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:23:06.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:23:06.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:23:06.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:23:06.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:23:06.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-06-03T07:23:07.415 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:ab946124:::obj15:head# 2024-06-03T07:23:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:23:07.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 16 % 2 2024-06-03T07:23:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-03T07:23:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-06-03T07:23:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:23:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:23:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:23:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:23:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:23:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-06-03T07:23:08.634 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:461f8b5e:::obj16:head# 2024-06-03T07:23:09.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:23:09.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 17 % 2 2024-06-03T07:23:09.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-03T07:23:09.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-06-03T07:23:09.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:23:09.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:23:09.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:23:09.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:23:09.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:23:09.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-06-03T07:23:09.814 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:24fc6e92:::obj17:head# 2024-06-03T07:23:10.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:23:10.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 18 % 2 2024-06-03T07:23:10.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-03T07:23:10.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-06-03T07:23:10.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:23:10.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:23:10.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:23:10.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:23:10.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:23:10.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-06-03T07:23:11.032 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:b9836a99:::obj18:head# 2024-06-03T07:23:11.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:23:11.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 19 % 2 2024-06-03T07:23:11.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-03T07:23:11.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-06-03T07:23:11.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:23:11.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:23:11.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:23:11.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:23:11.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:23:11.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-06-03T07:23:12.192 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:00b31b6c:::obj19:head# 2024-06-03T07:23:12.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-03T07:23:12.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 20 % 2 2024-06-03T07:23:12.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-03T07:23:12.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-06-03T07:23:12.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:23:12.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:23:12.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:23:12.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:23:12.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:23:12.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-06-03T07:23:13.428 INFO:tasks.workunit.client.0.smithi039.stdout:remove #1:24b5611b:::obj20:head# 2024-06-03T07:23:13.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:23:13.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:23:13.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:23:13.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:23:13.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:23:13.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:23:13.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:23:13.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:23:13.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:23:13.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:23:13.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:23:13.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:23:13.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:23:13.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:23:13.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:23:13.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:23:13.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:23:13.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:23:13.946 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:23:13.946 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:23:13.946 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:23:13.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:23:13.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:23:13.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:23:13.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:23:13.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:23:13.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:23:13.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:23:13.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:23:13.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:23:13.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:23:13.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T07:23:13.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:23:13.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:23:13.949 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T07:23:13.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:23:13.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:23:13.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:23:13.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:23:13.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:23:13.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:23:13.980 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:13.980+0000 7f0a03a07640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:23:13.982 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:13.983+0000 7f0a03a07640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:23:13.985 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:13.984+0000 7f0a03a07640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:23:14.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:23:14.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:23:14.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:23:14.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:23:14.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:23:14.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:14.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:23:14.249 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:23:14.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:14.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:23:14.498 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:14.499+0000 7f0a03a07640 -1 Falling back to public interface 2024-06-03T07:23:14.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:23:15.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:23:15.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:15.566 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:23:15.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:23:15.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:15.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:23:15.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:23:16.791 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:16.791+0000 7f0a03a07640 -1 osd.1 17 log_to_monitors true 2024-06-03T07:23:16.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:23:16.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:16.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:23:16.867 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:23:16.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:16.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:23:17.167 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 20 up_thru 17 down_at 18 last_clean_interval [10,17) [v2:127.0.0.1:6802/4222363774,v1:127.0.0.1:6803/4222363774] [v2:127.0.0.1:6804/4222363774,v1:127.0.0.1:6805/4222363774] exists,up 232342c9-70d4-4707-b07d-894e2f508ca2 2024-06-03T07:23:17.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:23:17.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:23:17.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:23:17.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:23:17.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:23:17.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:23:17.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:23:17.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:23:17.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:23:17.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:23:17.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:23:17.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:23:17.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:23:17.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:23:17.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:23:17.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:23:17.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:23:17.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:23:17.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:23:17.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:23:17.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:23:17.172 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T07:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:23:17.174 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T07:23:17.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:23:17.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:23:17.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:23:17.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:23:17.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:23:17.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:23:17.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:23:17.200 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:17.200+0000 7f862f81d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:23:17.202 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:17.202+0000 7f862f81d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:23:17.203 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:17.203+0000 7f862f81d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:23:17.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:23:17.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:23:17.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:23:17.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:23:17.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:23:17.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:17.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:23:17.533 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:23:17.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:17.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:23:17.723 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:17.724+0000 7f862f81d640 -1 Falling back to public interface 2024-06-03T07:23:17.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:23:18.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:23:18.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:18.867 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:23:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:23:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:18.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:23:19.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:23:20.032 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:20.032+0000 7f862f81d640 -1 osd.0 16 log_to_monitors true 2024-06-03T07:23:20.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:23:20.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:20.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:23:20.183 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:23:20.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:20.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:23:20.545 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6810/965056069,v1:127.0.0.1:6811/965056069] [v2:127.0.0.1:6812/965056069,v1:127.0.0.1:6813/965056069] exists,up cb8d7329-3611-489e-aea4-2ac5bcc34512 2024-06-03T07:23:20.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:23:20.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:23:20.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:23:20.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2024-06-03T07:23:20.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:23:20.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:23:20.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:23:20.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:23:20.547 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:23:20.547 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:23:20.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:23:20.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:23:20.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:23:20.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:23:20.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:23:20.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:23:20.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:23:20.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:23:20.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:23:20.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:23:20.955 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:23:20.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:23:20.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:23:20.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:23:21.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247810 2024-06-03T07:23:21.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247810 2024-06-03T07:23:21.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810' 2024-06-03T07:23:21.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:23:21.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:23:21.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345922 2024-06-03T07:23:21.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345922 2024-06-03T07:23:21.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810 1-85899345922' 2024-06-03T07:23:21.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:23:21.217 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247810 2024-06-03T07:23:21.217 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:23:21.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:23:21.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247810 2024-06-03T07:23:21.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:23:21.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247810 2024-06-03T07:23:21.220 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 98784247810 2024-06-03T07:23:21.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2024-06-03T07:23:21.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:23:21.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 98784247810 2024-06-03T07:23:21.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:23:22.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:23:22.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:23:22.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247810 -lt 98784247810 2024-06-03T07:23:22.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:23:22.865 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345922 2024-06-03T07:23:22.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:23:22.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:23:22.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345922 2024-06-03T07:23:22.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:23:22.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345922 2024-06-03T07:23:22.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345922' 2024-06-03T07:23:22.869 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 85899345922 2024-06-03T07:23:22.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:23:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345923 -lt 85899345922 2024-06-03T07:23:23.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:23:23.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:23:23.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:23:23.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T07:23:23.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:23:23.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:23:23.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:23:23.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:23:23.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:23:23.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:23:23.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:23:23.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T07:23:23.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:23:23.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:23:23.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:23:24.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T07:23:24.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:23:24.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:23:24.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: repair 1.0 2024-06-03T07:23:24.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-06-03T07:23:24.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-06-03T07:23:24.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T07:23:24.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:23:24.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:23:24.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T07:23:24.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T07:22:37.064981+0000 2024-06-03T07:23:24.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-06-03T07:23:24.769 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 1.0 on osd.1 to repair 2024-06-03T07:23:24.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-06-03T07:22:37.064981+0000 2024-06-03T07:23:24.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-03T07:23:24.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T07:22:37.064981+0000 2024-06-03T07:23:24.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T07:23:24.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:23:24.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:23:24.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T07:23:24.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T07:23:24.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:23:24.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:23:24.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T07:23:25.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:22:37.064981+0000 '>' 2024-06-03T07:22:37.064981+0000 2024-06-03T07:23:25.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:23:26.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:23:26.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:23:26.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T07:23:26.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T07:23:26.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:23:26.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:23:26.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T07:23:26.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:22:37.064981+0000 '>' 2024-06-03T07:22:37.064981+0000 2024-06-03T07:23:26.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:23:27.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:23:27.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:23:27.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T07:23:27.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T07:23:27.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:23:27.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:23:27.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T07:23:27.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:22:37.064981+0000 '>' 2024-06-03T07:22:37.064981+0000 2024-06-03T07:23:27.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:23:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:23:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:23:28.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T07:23:28.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T07:23:28.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:23:28.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:23:28.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T07:23:28.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:23:25.803319+0000 '>' 2024-06-03T07:22:37.064981+0000 2024-06-03T07:23:28.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:23:28.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: wait_for_clean 2024-06-03T07:23:28.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:23:28.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:23:28.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:23:28.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:23:28.961 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:23:28.961 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:23:28.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:23:28.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:23:28.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:23:29.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:23:29.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:23:29.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:23:29.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:23:29.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:23:29.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:23:29.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:23:29.347 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:23:29.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:23:29.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:23:29.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:23:29.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247812 2024-06-03T07:23:29.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247812 2024-06-03T07:23:29.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247812' 2024-06-03T07:23:29.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:23:29.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:23:29.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345925 2024-06-03T07:23:29.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345925 2024-06-03T07:23:29.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247812 1-85899345925' 2024-06-03T07:23:29.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:23:29.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247812 2024-06-03T07:23:29.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:23:29.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:23:29.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247812 2024-06-03T07:23:29.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:23:29.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247812 2024-06-03T07:23:29.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247812' 2024-06-03T07:23:29.568 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 98784247812 2024-06-03T07:23:29.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:23:29.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247811 -lt 98784247812 2024-06-03T07:23:29.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:23:30.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:23:30.859 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:23:31.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247813 -lt 98784247812 2024-06-03T07:23:31.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:23:31.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345925 2024-06-03T07:23:31.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:23:31.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:23:31.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345925 2024-06-03T07:23:31.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:23:31.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345925 2024-06-03T07:23:31.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345925' 2024-06-03T07:23:31.164 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 85899345925 2024-06-03T07:23:31.164 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:23:31.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345925 -lt 85899345925 2024-06-03T07:23:31.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:23:31.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:23:31.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:23:31.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T07:23:31.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:23:31.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:23:31.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:23:31.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:23:31.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:23:31.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:23:31.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:23:32.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T07:23:32.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:23:32.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:23:32.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:23:32.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T07:23:32.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:23:32.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:23:32.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: ceph pg dump pgs 2024-06-03T07:23:32.782 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T07:23:32.782 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-06-03T07:23:26.633572+0000 16'30 24:118 [1,0] 1 [1,0] 1 16'30 2024-06-03T07:23:25.803319+0000 16'30 2024-06-03T07:23:25.803319+0000 0 1 periodic scrub scheduled @ 2024-06-04T07:23:25.803319+0000 20 0 2024-06-03T07:23:32.783 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T07:23:32.783 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T07:23:32.783 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T07:23:32.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: flush_pg_stats 2024-06-03T07:23:32.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:23:32.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:23:33.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:23:33.080 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-06-03T07:23:33.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:23:33.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:23:33.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:23:33.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247814 2024-06-03T07:23:33.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247814 2024-06-03T07:23:33.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247814' 2024-06-03T07:23:33.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:23:33.189 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:23:33.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345927 2024-06-03T07:23:33.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345927 2024-06-03T07:23:33.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247814 1-85899345927' 2024-06-03T07:23:33.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:23:33.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247814 2024-06-03T07:23:33.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:23:33.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:23:33.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247814 2024-06-03T07:23:33.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:23:33.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247814 2024-06-03T07:23:33.300 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 98784247814 2024-06-03T07:23:33.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247814' 2024-06-03T07:23:33.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:23:33.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247813 -lt 98784247814 2024-06-03T07:23:33.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:23:34.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:23:34.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:23:34.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247814 -lt 98784247814 2024-06-03T07:23:34.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:23:34.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345927 2024-06-03T07:23:34.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:23:34.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:23:34.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345927 2024-06-03T07:23:34.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:23:34.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345927 2024-06-03T07:23:34.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345927' 2024-06-03T07:23:34.900 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 85899345927 2024-06-03T07:23:34.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:23:35.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345927 -lt 85899345927 2024-06-03T07:23:35.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: ceph pg 1.0 query 2024-06-03T07:23:35.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: jq .info.stats.stat_sum 2024-06-03T07:23:35.300 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T07:23:35.300 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 210, 2024-06-03T07:23:35.300 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 30, 2024-06-03T07:23:35.300 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-06-03T07:23:35.300 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 60, 2024-06-03T07:23:35.301 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-06-03T07:23:35.301 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-06-03T07:23:35.301 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 0, 2024-06-03T07:23:35.301 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-06-03T07:23:35.301 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-06-03T07:23:35.301 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 30, 2024-06-03T07:23:35.301 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-06-03T07:23:35.301 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-06-03T07:23:35.301 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-06-03T07:23:35.301 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 30, 2024-06-03T07:23:35.301 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 30, 2024-06-03T07:23:35.301 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-06-03T07:23:35.302 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-06-03T07:23:35.302 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-06-03T07:23:35.302 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 20, 2024-06-03T07:23:35.302 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 140, 2024-06-03T07:23:35.302 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-06-03T07:23:35.302 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-06-03T07:23:35.302 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-06-03T07:23:35.302 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-06-03T07:23:35.302 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-06-03T07:23:35.302 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-06-03T07:23:35.302 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-06-03T07:23:35.302 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-06-03T07:23:35.302 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-06-03T07:23:35.303 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-06-03T07:23:35.303 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-06-03T07:23:35.303 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-06-03T07:23:35.303 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-06-03T07:23:35.303 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-06-03T07:23:35.303 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-06-03T07:23:35.303 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-06-03T07:23:35.303 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-06-03T07:23:35.303 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-06-03T07:23:35.303 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-06-03T07:23:35.303 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 20 2024-06-03T07:23:35.303 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T07:23:35.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-06-03T07:23:35.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-06-03T07:23:35.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: COUNT=20 2024-06-03T07:23:35.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: test 20 = 20 2024-06-03T07:23:35.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-03T07:23:35.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-06-03T07:23:35.698 INFO:tasks.workunit.client.0.smithi039.stderr:dumped all 2024-06-03T07:23:35.709 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T07:23:35.709 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-06-03T07:23:35.709 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 20, 2024-06-03T07:23:35.709 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 85899345927, 2024-06-03T07:23:35.709 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 1, 2024-06-03T07:23:35.709 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 1, 2024-06-03T07:23:35.709 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 1, 2024-06-03T07:23:35.709 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 1, 2024-06-03T07:23:35.709 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 104857600, 2024-06-03T07:23:35.709 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 28552, 2024-06-03T07:23:35.709 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 320, 2024-06-03T07:23:35.710 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 4, 2024-06-03T07:23:35.710 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 28219, 2024-06-03T07:23:35.710 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 104829048, 2024-06-03T07:23:35.710 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-06-03T07:23:35.710 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 107374182400, 2024-06-03T07:23:35.710 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 107344945152, 2024-06-03T07:23:35.710 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-06-03T07:23:35.710 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 327680, 2024-06-03T07:23:35.710 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 53611, 2024-06-03T07:23:35.710 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-06-03T07:23:35.710 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-06-03T07:23:35.710 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-06-03T07:23:35.710 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 4398, 2024-06-03T07:23:35.710 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 28896978 2024-06-03T07:23:35.711 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T07:23:35.711 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [ 2024-06-03T07:23:35.711 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-06-03T07:23:35.711 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-06-03T07:23:35.711 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-06-03T07:23:35.711 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-06-03T07:23:35.711 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 10, 2024-06-03T07:23:35.711 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-06-03T07:23:35.711 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-06-03T07:23:35.711 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-06-03T07:23:35.711 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T07:23:35.711 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-06-03T07:23:35.711 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-06-03T07:23:35.712 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-06-03T07:23:35.712 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-06-03T07:23:35.712 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-06-03T07:23:35.712 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T07:23:35.712 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [] 2024-06-03T07:23:35.712 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T07:23:35.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-03T07:23:35.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-06-03T07:23:35.980 INFO:tasks.workunit.client.0.smithi039.stderr:dumped all 2024-06-03T07:23:35.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: COUNT=10 2024-06-03T07:23:35.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: expr 20 / 2 2024-06-03T07:23:35.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: test 10 = 10 2024-06-03T07:23:35.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-03T07:23:35.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-06-03T07:23:36.265 INFO:tasks.workunit.client.0.smithi039.stderr:dumped all 2024-06-03T07:23:36.276 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T07:23:36.276 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-06-03T07:23:36.276 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 23, 2024-06-03T07:23:36.276 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 98784247815, 2024-06-03T07:23:36.276 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 1, 2024-06-03T07:23:36.276 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 1, 2024-06-03T07:23:36.276 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 1, 2024-06-03T07:23:36.276 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 1, 2024-06-03T07:23:36.276 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 104857600, 2024-06-03T07:23:36.276 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 28552, 2024-06-03T07:23:36.277 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 320, 2024-06-03T07:23:36.277 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 4, 2024-06-03T07:23:36.277 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 28219, 2024-06-03T07:23:36.277 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 104829048, 2024-06-03T07:23:36.277 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-06-03T07:23:36.277 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 107374182400, 2024-06-03T07:23:36.277 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 107344945152, 2024-06-03T07:23:36.277 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-06-03T07:23:36.277 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 327680, 2024-06-03T07:23:36.277 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 53611, 2024-06-03T07:23:36.277 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-06-03T07:23:36.277 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-06-03T07:23:36.277 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-06-03T07:23:36.277 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 4401, 2024-06-03T07:23:36.278 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 28896975 2024-06-03T07:23:36.278 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T07:23:36.278 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [ 2024-06-03T07:23:36.278 INFO:tasks.workunit.client.0.smithi039.stdout: 1 2024-06-03T07:23:36.278 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-06-03T07:23:36.278 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-06-03T07:23:36.278 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-06-03T07:23:36.278 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 10, 2024-06-03T07:23:36.278 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-06-03T07:23:36.278 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-06-03T07:23:36.278 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-06-03T07:23:36.278 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T07:23:36.278 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-06-03T07:23:36.279 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-06-03T07:23:36.279 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-06-03T07:23:36.279 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-06-03T07:23:36.279 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-06-03T07:23:36.279 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T07:23:36.279 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [] 2024-06-03T07:23:36.279 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T07:23:36.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-03T07:23:36.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-06-03T07:23:36.549 INFO:tasks.workunit.client.0.smithi039.stderr:dumped all 2024-06-03T07:23:36.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: COUNT=10 2024-06-03T07:23:36.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: expr 20 / 2 2024-06-03T07:23:36.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: test 10 = 10 2024-06-03T07:23:36.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-03T07:23:36.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-06-03T07:23:36.834 INFO:tasks.workunit.client.0.smithi039.stderr:dumped all 2024-06-03T07:23:36.845 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T07:23:36.845 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-06-03T07:23:36.845 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-06-03T07:23:36.846 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 2, 2024-06-03T07:23:36.846 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 2, 2024-06-03T07:23:36.846 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 2, 2024-06-03T07:23:36.846 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 2, 2024-06-03T07:23:36.846 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 209715200, 2024-06-03T07:23:36.846 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 57104, 2024-06-03T07:23:36.846 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 640, 2024-06-03T07:23:36.846 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 8, 2024-06-03T07:23:36.846 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 56439, 2024-06-03T07:23:36.846 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 209658096, 2024-06-03T07:23:36.846 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-06-03T07:23:36.846 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 214748364800, 2024-06-03T07:23:36.846 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 214689890304, 2024-06-03T07:23:36.847 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-06-03T07:23:36.847 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 655360, 2024-06-03T07:23:36.847 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 107222, 2024-06-03T07:23:36.847 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-06-03T07:23:36.847 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-06-03T07:23:36.847 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-06-03T07:23:36.847 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 8799, 2024-06-03T07:23:36.847 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 57793953 2024-06-03T07:23:36.847 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T07:23:36.847 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-06-03T07:23:36.848 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-06-03T07:23:36.848 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-06-03T07:23:36.848 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 20, 2024-06-03T07:23:36.848 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-06-03T07:23:36.848 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-06-03T07:23:36.848 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-06-03T07:23:36.848 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T07:23:36.848 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-06-03T07:23:36.848 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-06-03T07:23:36.848 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-06-03T07:23:36.848 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-06-03T07:23:36.848 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-06-03T07:23:36.848 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T07:23:36.849 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-06-03T07:23:36.849 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-06-03T07:23:36.849 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T07:23:36.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-03T07:23:36.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-06-03T07:23:37.120 INFO:tasks.workunit.client.0.smithi039.stderr:dumped all 2024-06-03T07:23:37.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: COUNT=20 2024-06-03T07:23:37.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: test 20 = 20 2024-06-03T07:23:37.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-03T07:23:37.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:23:37.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:23:37.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:23:37.133 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:23:37.133 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:23:37.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:23:37.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:23:37.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:23:37.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:23:37.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:23:37.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:23:37.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:23:37.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:23:37.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:23:37.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:23:37.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:23:37.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:23:37.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:23:37.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:23:37.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:23:37.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:23:37.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:23:37.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:23:37.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:23:37.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:23:37.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:23:37.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:23:37.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:23:37.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-03T07:23:37.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-03T07:23:37.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-03T07:23:37.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-03T07:23:37.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:23:37.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T07:23:37.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:23:37.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:23:37.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:23:37.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:23:37.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:23:37.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:23:37.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:23:37.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:23:37.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:23:37.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:23:37.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:23:37.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:23:37.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:23:37.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:23:37.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:23:37.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:23:37.269 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:23:37.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:23:37.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T07:23:37.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:23:37.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:23:37.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:23:37.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:23:37.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:23:37.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:23:37.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:23:37.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-03T07:23:37.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T07:23:37.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:23:37.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:23:37.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69989 2024-06-03T07:23:37.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T07:23:37.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T07:23:37.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T07:23:37.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-03T07:23:37.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-06-03T07:23:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-06-03T07:23:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local poolname=testpool 2024-06-03T07:23:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local OSDS=3 2024-06-03T07:23:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OBJS=30 2024-06-03T07:23:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats_ec: local REPAIRS=26 2024-06-03T07:23:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local allow_overwrites=false 2024-06-03T07:23:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-06-03T07:23:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-03T07:23:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T07:23:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T07:23:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T07:23:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-03T07:23:37.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-03T07:23:37.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T07:23:37.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:23:37.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:23:37.662 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:23:37.662 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:23:37.662 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:23:37.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:23:37.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T07:23:37.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T07:23:37.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T07:23:37.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:23:37.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:23:37.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T07:23:37.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T07:23:37.848 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:23:37.848 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:23:37.848 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:23:37.848 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:23:37.848 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:23:37.848 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:23:37.848 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:23:37.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:23:37.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get fsid 2024-06-03T07:23:37.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T07:23:37.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T07:23:37.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T07:23:37.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T07:23:37.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T07:23:37.934 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T07:23:37.934 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T07:23:37.935 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T07:23:37.935 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T07:23:37.935 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:23:37.935 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:23:37.935 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69989/ceph-mon.a.asok 2024-06-03T07:23:37.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T07:23:37.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69989/ceph-mon.a.asok config get mon_host 2024-06-03T07:23:38.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-06-03T07:23:38.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-03T07:23:38.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T07:23:38.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T07:23:38.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T07:23:38.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-03T07:23:38.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T07:23:38.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T07:23:38.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:23:38.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:23:38.841 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:23:38.841 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:23:38.841 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:23:38.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:23:38.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:23:38.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T07:23:38.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T07:23:38.872 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: expr 3 - 1 2024-06-03T07:23:38.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: seq 0 2 2024-06-03T07:23:38.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T07:23:38.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:23:38.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:23:38.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:23:38.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T07:23:38.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:23:38.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:23:38.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:23:38.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:23:38.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:23:38.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:23:38.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:23:38.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:23:38.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:23:38.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:23:38.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:23:38.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:23:38.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:23:38.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:23:38.879 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:23:38.879 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:23:38.879 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:23:38.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:23:38.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:23:38.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:23:38.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:23:38.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:23:38.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:23:38.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:23:38.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:23:38.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:23:38.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:23:38.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:23:38.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:23:38.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T07:23:38.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:23:38.880 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:23:38.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=17688380-0558-484e-8571-52d151409fa1 2024-06-03T07:23:38.881 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 17688380-0558-484e-8571-52d151409fa1 2024-06-03T07:23:38.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 17688380-0558-484e-8571-52d151409fa1' 2024-06-03T07:23:38.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:23:38.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB6b11mYJycNRAAXidCDIYd1fpbf3pS3F/P6Q== 2024-06-03T07:23:38.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB6b11mYJycNRAAXidCDIYd1fpbf3pS3F/P6Q=="}' 2024-06-03T07:23:38.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 17688380-0558-484e-8571-52d151409fa1 -i td/osd-scrub-repair/0/new.json 2024-06-03T07:23:39.145 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:23:39.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-03T07:23:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQB6b11mYJycNRAAXidCDIYd1fpbf3pS3F/P6Q== --osd-uuid 17688380-0558-484e-8571-52d151409fa1 2024-06-03T07:23:39.191 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:39.192+0000 7feded706640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:23:39.205 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:39.205+0000 7feded706640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:23:39.206 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:39.206+0000 7feded706640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:23:39.206 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:39.207+0000 7feded706640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:23:39.206 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:39.207+0000 7feded706640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:23:39.206 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:39.207+0000 7feded706640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-03T07:23:39.305 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:39.305+0000 7feded706640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-03T07:23:45.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-03T07:23:45.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:23:45.125 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-06-03T07:23:45.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T07:23:45.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:23:45.506 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T07:23:45.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T07:23:45.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:23:45.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:23:45.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:23:45.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:23:45.535 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:45.536+0000 7f098160e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:23:45.537 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:45.538+0000 7f098160e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:23:45.539 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:45.539+0000 7f098160e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:23:45.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T07:23:45.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:23:45.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:23:45.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:23:45.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:23:45.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:45.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:23:45.800 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:23:45.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:45.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:23:46.051 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:46.052+0000 7f098160e640 -1 Falling back to public interface 2024-06-03T07:23:46.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:23:47.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:23:47.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:47.088 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:23:47.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:23:47.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:47.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:23:47.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:23:48.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:23:48.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:48.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:23:48.388 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:23:48.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:48.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:23:48.404 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:23:48.405+0000 7f098160e640 -1 osd.0 0 log_to_monitors true 2024-06-03T07:23:48.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:23:49.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:23:49.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:49.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:23:49.858 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:23:49.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:49.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:23:50.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:23:51.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:23:51.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:51.200 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:23:51.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:23:51.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:51.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:23:51.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:23:52.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:23:52.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:52.501 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:23:52.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:23:52.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:52.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:23:52.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:23:53.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:23:53.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:53.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:23:53.802 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:23:53.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:53.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:23:54.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:23:55.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:23:55.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:55.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:23:55.178 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:23:55.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:55.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:23:55.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:23:56.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:23:56.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:56.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:23:56.481 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:23:56.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:56.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:23:56.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:23:57.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:23:57.782 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:23:57.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:57.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:23:57.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:57.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:23:58.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:23:59.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:23:59.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:23:59.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:23:59.203 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:23:59.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:23:59.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:23:59.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:00.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:00.505 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:24:00.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:00.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:24:00.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:00.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:24:01.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:02.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:02.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:02.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:24:02.094 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:24:02.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:02.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:24:02.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:03.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:03.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:03.640 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:24:03.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:24:03.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:03.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:24:04.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:05.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:05.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:05.409 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:24:05.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:24:05.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:05.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:24:06.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:07.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:07.366 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:24:07.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:07.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:24:07.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:07.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:24:08.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:09.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:09.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:09.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:24:09.153 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:24:09.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:09.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:24:09.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:10.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:10.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:10.788 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:24:10.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:24:10.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:10.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:24:11.096 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/177450412,v1:127.0.0.1:6803/177450412] [v2:127.0.0.1:6804/177450412,v1:127.0.0.1:6805/177450412] exists,up 17688380-0558-484e-8571-52d151409fa1 2024-06-03T07:24:11.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:24:11.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:24:11.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:24:11.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T07:24:11.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:24:11.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:24:11.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:24:11.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T07:24:11.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:24:11.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:24:11.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:24:11.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:24:11.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:24:11.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:24:11.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:24:11.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:24:11.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:24:11.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:24:11.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:24:11.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:24:11.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:24:11.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:24:11.101 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:24:11.102 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:24:11.102 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:24:11.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:24:11.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:24:11.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:24:11.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:24:11.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:24:11.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:24:11.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:24:11.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:24:11.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:24:11.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:24:11.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:24:11.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:24:11.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T07:24:11.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:24:11.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:24:11.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4a060204-ac41-4fd6-a8d9-d7ef77cf2cea 2024-06-03T07:24:11.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4a060204-ac41-4fd6-a8d9-d7ef77cf2cea' 2024-06-03T07:24:11.104 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 4a060204-ac41-4fd6-a8d9-d7ef77cf2cea 2024-06-03T07:24:11.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:24:11.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCbb11mOiYhBxAAFREMD2nxFwuM66qm2uDd0g== 2024-06-03T07:24:11.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCbb11mOiYhBxAAFREMD2nxFwuM66qm2uDd0g=="}' 2024-06-03T07:24:11.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4a060204-ac41-4fd6-a8d9-d7ef77cf2cea -i td/osd-scrub-repair/1/new.json 2024-06-03T07:24:11.473 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:24:11.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-03T07:24:11.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCbb11mOiYhBxAAFREMD2nxFwuM66qm2uDd0g== --osd-uuid 4a060204-ac41-4fd6-a8d9-d7ef77cf2cea 2024-06-03T07:24:11.517 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:11.517+0000 7f5df0813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:24:11.574 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:11.575+0000 7f5df0813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:24:11.575 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:11.576+0000 7f5df0813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:24:11.576 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:11.576+0000 7f5df0813640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:24:11.576 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:11.576+0000 7f5df0813640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:24:11.576 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:11.576+0000 7f5df0813640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-03T07:24:11.651 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:11.651+0000 7f5df0813640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-03T07:24:17.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-03T07:24:17.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:24:17.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T07:24:17.915 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-06-03T07:24:17.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:24:18.298 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T07:24:18.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T07:24:18.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:24:18.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:24:18.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:24:18.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:24:18.327 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:18.327+0000 7efd7c56e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:24:18.329 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:18.329+0000 7efd7c56e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:24:18.330 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:18.330+0000 7efd7c56e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:24:18.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T07:24:18.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:24:18.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:24:18.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:24:18.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:24:18.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:18.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:24:18.592 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:24:18.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:18.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:18.856 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:18.857+0000 7efd7c56e640 -1 Falling back to public interface 2024-06-03T07:24:18.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:19.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:19.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:19.879 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:24:19.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:24:19.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:19.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:20.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:21.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:21.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:21.184 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:24:21.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:24:21.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:21.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:21.274 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:21.274+0000 7efd7c56e640 -1 osd.1 0 log_to_monitors true 2024-06-03T07:24:21.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:22.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:22.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:22.489 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:24:22.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:24:22.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:22.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:22.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:23.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:23.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:23.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:24:23.790 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:24:23.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:23.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:24.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:25.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:25.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:25.161 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:24:25.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:24:25.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:25.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:25.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:26.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:26.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:26.462 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:24:26.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:24:26.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:26.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:26.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:27.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:27.769 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:24:27.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:27.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:24:27.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:27.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:28.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:29.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:29.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:29.150 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:24:29.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:24:29.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:29.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:29.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:30.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:30.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:30.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:24:30.452 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:24:30.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:30.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:30.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:31.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:31.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:31.809 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:24:31.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:24:31.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:31.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:32.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:33.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:33.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:33.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:24:33.146 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:24:33.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:33.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:33.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:34.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:34.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:34.486 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:24:34.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:24:34.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:34.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:35.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:36.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:36.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:36.122 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:24:36.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:24:36.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:36.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:36.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:37.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:37.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:37.883 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:24:37.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:24:37.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:37.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:38.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:39.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:39.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:39.424 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:24:39.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:24:39.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:39.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:40.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:41.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:41.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:41.273 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:24:41.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:24:41.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:41.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:41.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:42.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:42.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:42.992 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:24:42.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:24:42.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:42.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:43.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:44.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:44.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:44.837 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T07:24:44.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T07:24:44.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:44.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:24:45.275 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1155299145,v1:127.0.0.1:6811/1155299145] [v2:127.0.0.1:6812/1155299145,v1:127.0.0.1:6813/1155299145] exists,up 4a060204-ac41-4fd6-a8d9-d7ef77cf2cea 2024-06-03T07:24:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:24:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:24:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:24:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T07:24:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:24:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-03T07:24:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T07:24:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T07:24:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T07:24:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-03T07:24:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-03T07:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-03T07:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T07:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T07:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T07:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:24:45.281 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:24:45.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:24:45.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:24:45.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:24:45.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T07:24:45.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T07:24:45.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:24:45.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:24:45.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:24:45.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:24:45.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:24:45.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:24:45.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T07:24:45.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T07:24:45.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-03T07:24:45.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T07:24:45.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b5843448-d2fc-40d4-8c64-67dc75bfe335 2024-06-03T07:24:45.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b5843448-d2fc-40d4-8c64-67dc75bfe335' 2024-06-03T07:24:45.283 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 b5843448-d2fc-40d4-8c64-67dc75bfe335 2024-06-03T07:24:45.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T07:24:45.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9b11muCHQERAAtAxmOTebchLEpxmbl2chYw== 2024-06-03T07:24:45.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9b11muCHQERAAtAxmOTebchLEpxmbl2chYw=="}' 2024-06-03T07:24:45.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b5843448-d2fc-40d4-8c64-67dc75bfe335 -i td/osd-scrub-repair/2/new.json 2024-06-03T07:24:45.661 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:24:45.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-03T07:24:45.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQC9b11muCHQERAAtAxmOTebchLEpxmbl2chYw== --osd-uuid b5843448-d2fc-40d4-8c64-67dc75bfe335 2024-06-03T07:24:45.704 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:45.703+0000 7f7356c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:24:45.745 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:45.744+0000 7f7356c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:24:45.747 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:45.746+0000 7f7356c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:24:45.747 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:45.746+0000 7f7356c0e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T07:24:45.747 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:45.746+0000 7f7356c0e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T07:24:45.747 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:45.746+0000 7f7356c0e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-03T07:24:45.830 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:45.829+0000 7f7356c0e640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-03T07:24:52.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-03T07:24:52.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T07:24:52.065 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-06-03T07:24:52.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T07:24:52.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T07:24:52.440 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-06-03T07:24:52.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T07:24:52.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:24:52.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T07:24:52.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T07:24:52.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T07:24:52.468 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:52.468+0000 7f82882ff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:24:52.471 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:52.470+0000 7f82882ff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:24:52.472 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:52.472+0000 7f82882ff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:24:52.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T07:24:52.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:24:52.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T07:24:52.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:24:52.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:24:52.773 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:24:52.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:52.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:24:52.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:52.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:24:52.987 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:52.987+0000 7f82882ff640 -1 Falling back to public interface 2024-06-03T07:24:53.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:54.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:54.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:54.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:24:54.067 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:24:54.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:54.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:24:54.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:55.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:55.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:55.368 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:24:55.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:24:55.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:55.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:24:55.393 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:24:55.393+0000 7f82882ff640 -1 osd.2 0 log_to_monitors true 2024-06-03T07:24:55.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:56.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:56.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:56.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:24:56.727 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:24:56.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:56.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:24:57.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:58.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:58.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:58.033 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-06-03T07:24:58.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T07:24:58.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:58.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:24:58.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:24:59.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:24:59.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:24:59.431 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-06-03T07:24:59.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T07:24:59.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:24:59.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:24:59.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:25:00.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:25:00.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:25:00.735 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-06-03T07:25:00.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T07:25:00.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:25:00.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:25:01.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:25:02.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:25:02.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:25:02.041 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-06-03T07:25:02.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T07:25:02.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:25:02.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:25:02.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:25:03.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:25:03.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:25:03.412 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-06-03T07:25:03.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T07:25:03.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:25:03.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:25:03.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:25:04.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:25:04.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:25:04.716 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-06-03T07:25:04.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T07:25:04.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:25:04.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:25:05.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:25:06.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:25:06.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:25:06.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T07:25:06.033 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-06-03T07:25:06.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:25:06.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:25:06.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:25:07.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:25:07.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:25:07.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T07:25:07.407 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-06-03T07:25:07.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:25:07.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:25:07.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:25:08.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:25:08.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:25:08.710 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-06-03T07:25:08.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T07:25:08.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:25:08.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:25:09.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:25:10.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:25:10.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:25:10.334 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-06-03T07:25:10.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T07:25:10.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:25:10.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:25:10.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:25:11.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:25:11.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:25:11.992 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-06-03T07:25:11.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T07:25:11.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:25:11.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:25:12.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:25:13.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:25:13.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:25:13.503 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-06-03T07:25:13.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T07:25:13.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:25:13.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:25:14.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:25:15.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:25:15.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:25:15.108 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-06-03T07:25:15.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T07:25:15.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:25:15.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:25:15.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:25:16.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:25:16.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:25:16.929 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-06-03T07:25:16.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T07:25:16.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:25:16.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:25:17.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:25:18.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:25:18.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:25:18.505 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-06-03T07:25:18.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T07:25:18.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:25:18.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T07:25:18.915 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3387742172,v1:127.0.0.1:6819/3387742172] [v2:127.0.0.1:6820/3387742172,v1:127.0.0.1:6821/3387742172] exists,up b5843448-d2fc-40d4-8c64-67dc75bfe335 2024-06-03T07:25:18.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:25:18.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:25:18.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:25:18.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-06-03T07:25:18.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=testpool 2024-06-03T07:25:18.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-03T07:25:18.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-06-03T07:25:18.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-03T07:25:18.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-06-03T07:25:19.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-06-03T07:25:19.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-06-03T07:25:20.143 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'testpool' created 2024-06-03T07:25:20.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T07:25:21.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-06-03T07:25:21.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-03T07:25:21.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:25:21.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:25:21.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:25:21.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:25:21.158 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:25:21.158 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:25:21.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:25:21.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:25:21.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:25:21.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:25:21.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:25:21.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:25:21.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:25:21.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:25:21.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:25:21.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:25:21.703 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:25:21.703 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:25:21.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:25:21.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:25:21.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:25:21.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-06-03T07:25:21.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-06-03T07:25:21.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-06-03T07:25:21.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:25:21.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:25:22.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-06-03T07:25:22.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-06-03T07:25:22.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2024-06-03T07:25:22.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:25:22.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:25:22.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-06-03T07:25:22.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-06-03T07:25:22.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509447' 2024-06-03T07:25:22.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:25:22.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-06-03T07:25:22.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:25:22.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:25:22.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-06-03T07:25:22.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:25:22.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-06-03T07:25:22.166 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836495 2024-06-03T07:25:22.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-06-03T07:25:22.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:25:22.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-06-03T07:25:22.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:25:23.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:25:23.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:25:23.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836495 2024-06-03T07:25:23.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:25:23.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-06-03T07:25:23.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:25:23.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:25:23.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-06-03T07:25:23.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:25:23.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-06-03T07:25:23.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-06-03T07:25:23.762 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672974 2024-06-03T07:25:23.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:25:24.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-06-03T07:25:24.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:25:24.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-06-03T07:25:24.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:25:24.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:25:24.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-06-03T07:25:24.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:25:24.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-06-03T07:25:24.088 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509447 2024-06-03T07:25:24.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-06-03T07:25:24.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:25:24.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-06-03T07:25:24.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:25:24.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:25:24.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:25:24.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T07:25:24.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:25:24.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:25:24.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:25:24.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:25:24.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:25:24.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:25:24.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:25:25.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T07:25:25.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:25:25.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:25:25.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:25:25.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T07:25:25.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:25:25.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:25:25.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-03T07:25:25.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats_ec: local payload=ABCDEF 2024-06-03T07:25:25.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: echo ABCDEF 2024-06-03T07:25:25.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: seq 1 30 2024-06-03T07:25:25.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:25.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:25.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:25.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:25.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:25.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:26.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:26.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:26.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:26.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:26.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:26.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:26.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:26.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:27.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:27.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:27.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:27.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:27.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:27.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:27.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:27.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:27.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:27.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:28.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:28.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:28.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:28.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:28.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:28.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:28.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:28.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:29.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:29.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:29.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:29.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:29.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:29.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:29.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:29.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:30.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:30.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:30.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:30.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:30.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:30.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:30.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:30.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:31.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:31.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:31.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:31.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:31.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:31.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:31.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:31.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:31.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:31.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:32.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-03T07:25:32.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-06-03T07:25:32.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-06-03T07:25:32.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-03T07:25:32.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-06-03T07:25:32.331 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-06-03T07:25:32.331 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T07:25:32.331 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-03T07:25:32.332 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-03T07:25:32.332 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T07:25:32.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T07:25:32.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-06-03T07:25:32.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T07:25:32.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: local other=0 2024-06-03T07:25:32.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_pg testpool obj1 2024-06-03T07:25:32.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-03T07:25:32.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-06-03T07:25:32.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-06-03T07:25:32.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T07:25:33.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local pgid=1.0 2024-06-03T07:25:33.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_primary testpool obj1 2024-06-03T07:25:33.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-03T07:25:33.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-03T07:25:33.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-03T07:25:33.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T07:25:33.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local primary=1 2024-06-03T07:25:33.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:758: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-03T07:25:33.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:25:33.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:25:33.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:25:33.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:25:33.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:25:33.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:25:33.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-03T07:25:33.821 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:25:33.821 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:25:33.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:25:33.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:25:33.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:25:34.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:25:34.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: seq 1 26 2024-06-03T07:25:34.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:34.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 1 % 2 2024-06-03T07:25:34.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-03T07:25:34.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-06-03T07:25:34.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:34.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:34.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:25:34.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:34.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:25:34.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-06-03T07:25:35.524 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-06-03T07:25:36.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:36.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 2 % 2 2024-06-03T07:25:36.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-03T07:25:36.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-06-03T07:25:36.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:36.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:36.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:25:36.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:36.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:25:36.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-06-03T07:25:37.259 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#1:104778fc:::obj2:head# 2024-06-03T07:25:37.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:37.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 3 % 2 2024-06-03T07:25:37.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-03T07:25:37.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-06-03T07:25:37.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:37.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:37.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:25:37.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:37.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:25:37.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-06-03T07:25:38.550 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-06-03T07:25:39.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:39.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 4 % 2 2024-06-03T07:25:39.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-03T07:25:39.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-06-03T07:25:39.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:39.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:39.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:25:39.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:39.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:25:39.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-06-03T07:25:39.920 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-06-03T07:25:40.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:40.438 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 5 % 2 2024-06-03T07:25:40.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-03T07:25:40.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-06-03T07:25:40.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:40.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:40.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:25:40.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:40.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:25:40.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-06-03T07:25:41.189 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#1:c52c9666:::obj5:head# 2024-06-03T07:25:41.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:41.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 6 % 2 2024-06-03T07:25:41.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-03T07:25:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-06-03T07:25:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:25:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:25:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-06-03T07:25:42.417 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#1:233a42c1:::obj6:head# 2024-06-03T07:25:42.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:42.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 7 % 2 2024-06-03T07:25:42.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-03T07:25:42.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-06-03T07:25:42.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:42.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:42.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:25:42.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:42.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:25:42.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-06-03T07:25:43.602 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#1:9a09113a:::obj7:head# 2024-06-03T07:25:44.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:44.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 8 % 2 2024-06-03T07:25:44.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-03T07:25:44.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-06-03T07:25:44.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:44.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:44.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:25:44.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:44.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:25:44.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-06-03T07:25:44.874 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#1:a557efb1:::obj8:head# 2024-06-03T07:25:45.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:45.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 9 % 2 2024-06-03T07:25:45.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-03T07:25:45.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-06-03T07:25:45.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:45.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:45.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:25:45.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:45.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:25:45.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-06-03T07:25:46.124 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#1:fe351e27:::obj9:head# 2024-06-03T07:25:46.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:46.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 10 % 2 2024-06-03T07:25:46.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-03T07:25:46.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-06-03T07:25:46.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:46.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:46.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:25:46.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:46.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:25:46.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-06-03T07:25:47.385 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#1:d1337354:::obj10:head# 2024-06-03T07:25:47.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:47.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 11 % 2 2024-06-03T07:25:47.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-03T07:25:47.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-06-03T07:25:47.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:47.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:47.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:25:47.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:47.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:25:47.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-06-03T07:25:48.629 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-06-03T07:25:49.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:49.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 12 % 2 2024-06-03T07:25:49.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-03T07:25:49.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-06-03T07:25:49.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:49.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:49.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:25:49.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:49.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:25:49.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-06-03T07:25:50.216 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-06-03T07:25:50.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:50.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 13 % 2 2024-06-03T07:25:50.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-03T07:25:50.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-06-03T07:25:50.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:50.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:50.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:25:50.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:50.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:25:50.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-06-03T07:25:51.619 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-06-03T07:25:52.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:52.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 14 % 2 2024-06-03T07:25:52.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-03T07:25:52.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-06-03T07:25:52.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:52.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:52.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:25:52.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:52.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:25:52.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-06-03T07:25:52.754 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#1:e0a44829:::obj14:head# 2024-06-03T07:25:53.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:53.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 15 % 2 2024-06-03T07:25:53.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-03T07:25:53.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-06-03T07:25:53.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:53.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:53.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:25:53.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:53.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:25:53.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-06-03T07:25:53.873 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#1:ab946124:::obj15:head# 2024-06-03T07:25:54.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:54.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 16 % 2 2024-06-03T07:25:54.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-03T07:25:54.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-06-03T07:25:54.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:54.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:54.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:25:54.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:54.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:25:54.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-06-03T07:25:55.049 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-06-03T07:25:55.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:55.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 17 % 2 2024-06-03T07:25:55.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-03T07:25:55.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-06-03T07:25:55.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:55.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:55.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:25:55.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:55.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:25:55.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-06-03T07:25:56.344 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-06-03T07:25:56.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:56.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 18 % 2 2024-06-03T07:25:56.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-03T07:25:56.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-06-03T07:25:56.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:56.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:56.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:25:56.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:56.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:25:56.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-06-03T07:25:57.505 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#1:b9836a99:::obj18:head# 2024-06-03T07:25:58.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:58.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 19 % 2 2024-06-03T07:25:58.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-03T07:25:58.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-06-03T07:25:58.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:58.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:58.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:25:58.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:58.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:25:58.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-06-03T07:25:58.674 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-06-03T07:25:59.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:25:59.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 20 % 2 2024-06-03T07:25:59.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-03T07:25:59.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-06-03T07:25:59.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:25:59.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:25:59.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:25:59.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:25:59.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:25:59.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-06-03T07:25:59.909 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#1:24b5611b:::obj20:head# 2024-06-03T07:26:00.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:26:00.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 21 % 2 2024-06-03T07:26:00.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-03T07:26:00.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-06-03T07:26:00.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:26:00.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:26:00.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:26:00.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:26:00.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:26:00.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-06-03T07:26:01.220 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-06-03T07:26:01.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:26:01.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 22 % 2 2024-06-03T07:26:01.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-03T07:26:01.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-06-03T07:26:01.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:26:01.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:26:01.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:26:01.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:26:01.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:26:01.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-06-03T07:26:02.447 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#1:c0c49def:::obj22:head# 2024-06-03T07:26:02.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:26:02.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 23 % 2 2024-06-03T07:26:02.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-03T07:26:02.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-06-03T07:26:02.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:26:02.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:26:02.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:26:02.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:26:02.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:26:02.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-06-03T07:26:03.683 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#1:242c40e0:::obj23:head# 2024-06-03T07:26:04.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:26:04.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 24 % 2 2024-06-03T07:26:04.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-03T07:26:04.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-06-03T07:26:04.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:26:04.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:26:04.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:26:04.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:26:04.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:26:04.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-06-03T07:26:04.885 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#1:7eacede5:::obj24:head# 2024-06-03T07:26:05.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:26:05.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 25 % 2 2024-06-03T07:26:05.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-03T07:26:05.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-06-03T07:26:05.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:26:05.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:26:05.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-03T07:26:05.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:26:05.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:26:05.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-06-03T07:26:06.163 INFO:tasks.workunit.client.0.smithi039.stdout:remove 0#1:7109478f:::obj25:head# 2024-06-03T07:26:06.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-03T07:26:06.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 26 % 2 2024-06-03T07:26:06.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-03T07:26:06.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-06-03T07:26:06.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-03T07:26:06.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T07:26:06.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T07:26:06.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T07:26:06.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:26:06.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-06-03T07:26:07.440 INFO:tasks.workunit.client.0.smithi039.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-06-03T07:26:07.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:26:07.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:26:07.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:26:07.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T07:26:07.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:26:07.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-03T07:26:07.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:26:07.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:26:07.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:26:07.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:26:07.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-03T07:26:07.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-03T07:26:07.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:26:07.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:26:07.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:26:07.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:26:07.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:26:07.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:26:07.961 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:26:07.961 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:26:07.961 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:26:07.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:26:07.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:26:07.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:26:07.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:26:07.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:26:07.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:26:07.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:26:07.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:26:07.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:26:07.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:26:07.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T07:26:07.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-03T07:26:07.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T07:26:07.964 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-06-03T07:26:07.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:26:07.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-03T07:26:07.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T07:26:07.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:26:07.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:26:07.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:26:07.995 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:26:07.995+0000 7eff53b1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:26:07.997 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:26:07.998+0000 7eff53b1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:26:07.999 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:26:07.999+0000 7eff53b1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:26:08.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T07:26:08.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:26:08.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T07:26:08.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:26:08.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:26:08.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:26:08.271 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:26:08.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:26:08.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:26:08.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:26:08.514 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:26:08.514+0000 7eff53b1c640 -1 Falling back to public interface 2024-06-03T07:26:08.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:26:09.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:26:09.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:26:09.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:26:09.565 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:26:09.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:26:09.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:26:09.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:26:10.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:26:10.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:26:10.864 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:26:10.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:26:10.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:26:10.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:26:11.011 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:26:11.011+0000 7eff53b1c640 -1 osd.1 22 log_to_monitors true 2024-06-03T07:26:11.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:26:12.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:26:12.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:26:12.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:26:12.171 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:26:12.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:26:12.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T07:26:12.478 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [10,22) [v2:127.0.0.1:6802/3345387482,v1:127.0.0.1:6803/3345387482] [v2:127.0.0.1:6804/3345387482,v1:127.0.0.1:6805/3345387482] exists,up 4a060204-ac41-4fd6-a8d9-d7ef77cf2cea 2024-06-03T07:26:12.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:26:12.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:26:12.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:26:12.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:26:12.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-03T07:26:12.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T07:26:12.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T07:26:12.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T07:26:12.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-03T07:26:12.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-03T07:26:12.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T07:26:12.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T07:26:12.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T07:26:12.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-03T07:26:12.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-03T07:26:12.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T07:26:12.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T07:26:12.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-03T07:26:12.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T07:26:12.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T07:26:12.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T07:26:12.483 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T07:26:12.483 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:26:12.483 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:26:12.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:26:12.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' 2024-06-03T07:26:12.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T07:26:12.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-03T07:26:12.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-03T07:26:12.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T07:26:12.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T07:26:12.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T07:26:12.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T07:26:12.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T07:26:12.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-03T07:26:12.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-03T07:26:12.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T07:26:12.484 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-06-03T07:26:12.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=578eade9-3a31-4e01-8b11-ea072a6a7a75 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69989/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-03T07:26:12.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-03T07:26:12.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T07:26:12.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T07:26:12.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T07:26:12.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T07:26:12.509 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:26:12.510+0000 7f7f1761c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:26:12.512 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:26:12.512+0000 7f7f1761c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:26:12.513 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:26:12.513+0000 7f7f1761c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T07:26:12.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T07:26:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T07:26:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T07:26:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T07:26:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T07:26:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:26:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T07:26:12.774 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-06-03T07:26:12.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:26:12.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:26:13.031 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:26:13.031+0000 7f7f1761c640 -1 Falling back to public interface 2024-06-03T07:26:13.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:26:14.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:26:14.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:26:14.063 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-06-03T07:26:14.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T07:26:14.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:26:14.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:26:14.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:26:15.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:26:15.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:26:15.366 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-06-03T07:26:15.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T07:26:15.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:26:15.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:26:15.413 INFO:tasks.workunit.client.0.smithi039.stderr:2024-06-03T07:26:15.413+0000 7f7f1761c640 -1 osd.0 21 log_to_monitors true 2024-06-03T07:26:15.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T07:26:16.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T07:26:16.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T07:26:16.686 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-06-03T07:26:16.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T07:26:16.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T07:26:16.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T07:26:16.986 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/878969045,v1:127.0.0.1:6811/878969045] [v2:127.0.0.1:6812/878969045,v1:127.0.0.1:6813/878969045] exists,up 17688380-0558-484e-8571-52d151409fa1 2024-06-03T07:26:16.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T07:26:16.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T07:26:16.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T07:26:16.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2024-06-03T07:26:16.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:26:16.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:26:16.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:26:16.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:26:16.988 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:26:16.988 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:26:16.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:26:16.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:26:16.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:26:17.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:26:17.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:26:17.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:26:17.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:26:17.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:26:17.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:26:17.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:26:17.377 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:26:17.377 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:26:17.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:26:17.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:26:17.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:26:17.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-06-03T07:26:17.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-06-03T07:26:17.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-06-03T07:26:17.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:26:17.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:26:17.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149699 2024-06-03T07:26:17.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149699 2024-06-03T07:26:17.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-111669149699' 2024-06-03T07:26:17.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:26:17.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:26:17.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-06-03T07:26:17.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-06-03T07:26:17.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-111669149699 2-64424509459' 2024-06-03T07:26:17.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:26:17.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-06-03T07:26:17.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:26:17.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:26:17.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-06-03T07:26:17.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:26:17.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-06-03T07:26:17.706 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 124554051586 2024-06-03T07:26:17.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-06-03T07:26:17.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:26:17.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-06-03T07:26:17.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:26:18.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:26:18.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:26:19.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-06-03T07:26:19.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:26:19.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149699 2024-06-03T07:26:19.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:26:19.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:26:19.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149699 2024-06-03T07:26:19.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:26:19.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149699 2024-06-03T07:26:19.299 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 111669149699 2024-06-03T07:26:19.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149699' 2024-06-03T07:26:19.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:26:19.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149699 -lt 111669149699 2024-06-03T07:26:19.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:26:19.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-06-03T07:26:19.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:26:19.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:26:19.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-06-03T07:26:19.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:26:19.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-06-03T07:26:19.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-06-03T07:26:19.591 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509459 2024-06-03T07:26:19.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:26:19.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509459 2024-06-03T07:26:19.880 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:26:19.880 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:26:19.880 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:26:20.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T07:26:20.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:26:20.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:26:20.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:26:20.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:26:20.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:26:20.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:26:20.240 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:26:20.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T07:26:20.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:26:20.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:26:20.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:26:20.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T07:26:20.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:26:20.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:26:20.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: repair 1.0 2024-06-03T07:26:20.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-06-03T07:26:20.889 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-06-03T07:26:20.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T07:26:20.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:26:20.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:26:20.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T07:26:21.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T07:25:20.026238+0000 2024-06-03T07:26:21.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-06-03T07:26:21.450 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-06-03T07:26:21.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-06-03T07:25:20.026238+0000 2024-06-03T07:26:21.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-03T07:26:21.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T07:25:20.026238+0000 2024-06-03T07:26:21.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T07:26:21.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T07:26:21.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:26:21.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T07:26:21.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T07:26:21.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:26:21.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:26:21.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T07:26:21.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:25:20.026238+0000 '>' 2024-06-03T07:25:20.026238+0000 2024-06-03T07:26:21.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:26:22.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:26:22.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:26:22.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T07:26:22.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T07:26:22.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:26:22.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:26:22.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T07:26:23.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:25:20.026238+0000 '>' 2024-06-03T07:25:20.026238+0000 2024-06-03T07:26:23.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:26:24.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:26:24.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:26:24.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T07:26:24.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T07:26:24.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:26:24.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:26:24.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T07:26:24.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:25:20.026238+0000 '>' 2024-06-03T07:25:20.026238+0000 2024-06-03T07:26:24.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:26:25.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:26:25.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:26:25.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T07:26:25.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T07:26:25.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:26:25.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:26:25.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T07:26:25.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:25:20.026238+0000 '>' 2024-06-03T07:25:20.026238+0000 2024-06-03T07:26:25.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:26:26.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:26:26.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:26:26.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T07:26:26.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T07:26:26.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:26:26.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:26:26.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T07:26:26.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:25:20.026238+0000 '>' 2024-06-03T07:25:20.026238+0000 2024-06-03T07:26:26.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T07:26:27.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T07:26:27.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T07:26:27.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T07:26:27.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T07:26:27.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T07:26:27.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T07:26:27.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T07:26:28.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T07:26:22.119473+0000 '>' 2024-06-03T07:25:20.026238+0000 2024-06-03T07:26:28.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T07:26:28.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: wait_for_clean 2024-06-03T07:26:28.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T07:26:28.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T07:26:28.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T07:26:28.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T07:26:28.250 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T07:26:28.250 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T07:26:28.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T07:26:28.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T07:26:28.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T07:26:28.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T07:26:28.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T07:26:28.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T07:26:28.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T07:26:28.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:26:28.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:26:28.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:26:28.643 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:26:28.643 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:26:28.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:26:28.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:26:28.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:26:28.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-06-03T07:26:28.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-06-03T07:26:28.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589' 2024-06-03T07:26:28.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:26:28.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:26:28.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-06-03T07:26:28.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-06-03T07:26:28.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-111669149702' 2024-06-03T07:26:28.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:26:28.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:26:28.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-06-03T07:26:28.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-06-03T07:26:28.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-111669149702 2-64424509463' 2024-06-03T07:26:28.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:26:28.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051589 2024-06-03T07:26:28.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:26:28.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:26:28.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051589 2024-06-03T07:26:28.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:26:28.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-06-03T07:26:28.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-06-03T07:26:28.972 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 124554051589 2024-06-03T07:26:28.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:26:29.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051589 2024-06-03T07:26:29.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:26:29.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149702 2024-06-03T07:26:29.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:26:29.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:26:29.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149702 2024-06-03T07:26:29.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:26:29.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-06-03T07:26:29.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149702' 2024-06-03T07:26:29.274 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 111669149702 2024-06-03T07:26:29.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:26:29.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149702 -lt 111669149702 2024-06-03T07:26:29.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:26:29.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-06-03T07:26:29.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:26:29.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:26:29.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-06-03T07:26:29.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:26:29.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-06-03T07:26:29.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-06-03T07:26:29.566 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509463 2024-06-03T07:26:29.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:26:29.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-06-03T07:26:29.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T07:26:29.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:26:29.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:26:30.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-03T07:26:30.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T07:26:30.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T07:26:30.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T07:26:30.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T07:26:30.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T07:26:30.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T07:26:30.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T07:26:30.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-03T07:26:30.507 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T07:26:30.507 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T07:26:30.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T07:26:30.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-03T07:26:30.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T07:26:30.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T07:26:30.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: ceph pg dump pgs 2024-06-03T07:26:31.226 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-03T07:26:31.227 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-06-03T07:26:23.142604+0000 21'30 30:130 [1,0,2] 1 [1,0,2] 1 21'30 2024-06-03T07:26:22.119473+0000 21'30 2024-06-03T07:26:22.119473+0000 0 1 periodic scrub scheduled @ 2024-06-04T07:26:22.119473+0000 17 0 2024-06-03T07:26:31.227 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-06-03T07:26:31.227 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-03T07:26:31.227 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-06-03T07:26:31.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: flush_pg_stats 2024-06-03T07:26:31.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T07:26:31.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T07:26:31.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T07:26:31.528 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-06-03T07:26:31.528 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-06-03T07:26:31.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T07:26:31.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:26:31.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T07:26:31.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051591 2024-06-03T07:26:31.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051591 2024-06-03T07:26:31.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591' 2024-06-03T07:26:31.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:26:31.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T07:26:31.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149704 2024-06-03T07:26:31.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149704 2024-06-03T07:26:31.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591 1-111669149704' 2024-06-03T07:26:31.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T07:26:31.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T07:26:31.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-06-03T07:26:31.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-06-03T07:26:31.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591 1-111669149704 2-64424509464' 2024-06-03T07:26:31.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:26:31.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051591 2024-06-03T07:26:31.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:26:31.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T07:26:31.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051591 2024-06-03T07:26:31.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:26:31.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051591 2024-06-03T07:26:31.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051591' 2024-06-03T07:26:31.858 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 124554051591 2024-06-03T07:26:31.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:26:32.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051590 -lt 124554051591 2024-06-03T07:26:32.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T07:26:33.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T07:26:33.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T07:26:33.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051591 -lt 124554051591 2024-06-03T07:26:33.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:26:33.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149704 2024-06-03T07:26:33.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:26:33.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T07:26:33.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149704 2024-06-03T07:26:33.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:26:33.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149704 2024-06-03T07:26:33.454 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 111669149704 2024-06-03T07:26:33.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149704' 2024-06-03T07:26:33.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T07:26:33.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149704 -lt 111669149704 2024-06-03T07:26:33.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T07:26:33.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-06-03T07:26:33.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T07:26:33.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T07:26:33.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-06-03T07:26:33.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T07:26:33.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-06-03T07:26:33.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-06-03T07:26:33.752 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509464 2024-06-03T07:26:33.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T07:26:34.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-06-03T07:26:34.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: ceph pg 1.0 query 2024-06-03T07:26:34.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-06-03T07:26:34.175 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T07:26:34.175 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 210, 2024-06-03T07:26:34.175 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 30, 2024-06-03T07:26:34.176 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-06-03T07:26:34.176 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 90, 2024-06-03T07:26:34.176 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-06-03T07:26:34.176 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-06-03T07:26:34.176 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 0, 2024-06-03T07:26:34.176 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-06-03T07:26:34.176 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-06-03T07:26:34.176 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 30, 2024-06-03T07:26:34.176 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-06-03T07:26:34.176 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-06-03T07:26:34.176 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-06-03T07:26:34.176 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 30, 2024-06-03T07:26:34.176 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 30, 2024-06-03T07:26:34.176 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-06-03T07:26:34.176 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 26, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 182, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-06-03T07:26:34.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-06-03T07:26:34.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-06-03T07:26:34.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-06-03T07:26:34.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-06-03T07:26:34.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-06-03T07:26:34.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-06-03T07:26:34.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-06-03T07:26:34.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 26 2024-06-03T07:26:34.178 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T07:26:34.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-06-03T07:26:34.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-06-03T07:26:34.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: COUNT=26 2024-06-03T07:26:34.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: test 26 = 26 2024-06-03T07:26:34.288 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: expr 3 - 1 2024-06-03T07:26:34.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: seq 0 2 2024-06-03T07:26:34.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T07:26:34.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-06-03T07:26:34.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-06-03T07:26:34.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-06-03T07:26:34.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-06-03T07:26:34.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-06-03T07:26:34.564 INFO:tasks.workunit.client.0.smithi039.stderr:dumped all 2024-06-03T07:26:34.575 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-06-03T07:26:34.575 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-06-03T07:26:34.576 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 29, 2024-06-03T07:26:34.576 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 124554051591, 2024-06-03T07:26:34.576 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 1, 2024-06-03T07:26:34.576 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 1, 2024-06-03T07:26:34.576 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 1, 2024-06-03T07:26:34.576 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 1, 2024-06-03T07:26:34.576 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 104857600, 2024-06-03T07:26:34.576 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 28472, 2024-06-03T07:26:34.576 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 368, 2024-06-03T07:26:34.576 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 4, 2024-06-03T07:26:34.576 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 28091, 2024-06-03T07:26:34.576 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 104829128, 2024-06-03T07:26:34.576 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-06-03T07:26:34.576 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 107374182400, 2024-06-03T07:26:34.577 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 107345027072, 2024-06-03T07:26:34.577 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-06-03T07:26:34.577 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 376832, 2024-06-03T07:26:34.577 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 203494, 2024-06-03T07:26:34.577 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-06-03T07:26:34.577 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-06-03T07:26:34.577 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-06-03T07:26:34.577 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 4402, 2024-06-03T07:26:34.577 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 28765902 2024-06-03T07:26:34.577 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T07:26:34.577 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [ 2024-06-03T07:26:34.577 INFO:tasks.workunit.client.0.smithi039.stdout: 1, 2024-06-03T07:26:34.577 INFO:tasks.workunit.client.0.smithi039.stdout: 2 2024-06-03T07:26:34.577 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-06-03T07:26:34.578 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-06-03T07:26:34.578 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-06-03T07:26:34.578 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 26, 2024-06-03T07:26:34.578 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-06-03T07:26:34.578 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-06-03T07:26:34.578 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-06-03T07:26:34.578 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T07:26:34.578 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-06-03T07:26:34.578 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-06-03T07:26:34.578 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-06-03T07:26:34.578 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-06-03T07:26:34.578 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-06-03T07:26:34.578 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-06-03T07:26:34.579 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [] 2024-06-03T07:26:34.579 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-06-03T07:26:34.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-06-03T07:26:34.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-06-03T07:26:34.850 INFO:tasks.workunit.client.0.smithi039.stderr:dumped all 2024-06-03T07:26:34.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=26 2024-06-03T07:26:34.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 26 = 13 2024-06-03T07:26:34.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: return 1 2024-06-03T07:26:34.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: return 1 2024-06-03T07:26:34.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-06-03T07:26:34.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-repair 1 2024-06-03T07:26:34.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-03T07:26:34.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-06-03T07:26:34.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-03T07:26:34.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T07:26:34.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T07:26:34.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T07:26:34.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T07:26:34.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T07:26:34.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T07:26:34.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T07:26:34.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T07:26:34.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T07:26:34.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T07:26:34.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T07:26:34.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T07:26:34.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:26:34.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T07:26:34.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T07:26:34.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T07:26:34.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:26:34.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-06-03T07:26:34.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-06-03T07:26:34.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-06-03T07:26:34.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-scrub-repair/mgr.x.log td/osd-scrub-repair/mon.a.log td/osd-scrub-repair/osd.0.log td/osd-scrub-repair/osd.1.log td/osd-scrub-repair/osd.2.log /home/ubuntu/cephtest/archive/log 2024-06-03T07:26:34.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-03T07:26:34.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T07:26:34.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T07:26:34.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69989 2024-06-03T07:26:34.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69989 2024-06-03T07:26:34.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T07:26:34.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T07:26:34.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T07:26:34.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-06-03T07:26:34.993 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-06-03T07:26:34.993 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-06-03T07:26:35.336 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-06-03T07:26:35.555 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=169a26c25a1c4a16b02980d78a84d1ca Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-06-03T07:26:35.558 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-03T07:26:35.568 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-03T07:26:35.569 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-03T07:26:35.618 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-03T07:26:35.626 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-03T07:26:35.627 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-03T07:26:35.654 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2024-06-03T07:26:35.659 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-03T07:26:35.659 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2024-06-03T07:26:35.659 INFO:teuthology.orchestra.run.smithi039.stdout:^* hv01.front.sepia.ceph.com 3 7 377 127 +54us[ +54us] +/- 44ms 2024-06-03T07:26:35.659 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv02.front.sepia.ceph.com 3 9 377 317 -839us[ -833us] +/- 61ms 2024-06-03T07:26:35.660 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 +1104us[+1104us] +/- 83ms 2024-06-03T07:26:35.660 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T07:26:35.660 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-03T07:26:35.668 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-03T07:26:35.669 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-03T07:26:35.677 DEBUG:teuthology.orchestra.run.smithi039:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-03T07:26:35.758 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-03T07:26:35.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T07:26:35.786 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-03T07:26:35.794 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-03T07:26:35.802 INFO:teuthology.task.internal:Duration was 5698.407599 seconds 2024-06-03T07:26:35.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-03T07:26:35.811 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-03T07:26:35.811 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-03T07:26:35.866 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-03T07:26:36.334 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-03T07:26:36.334 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2024-06-03T07:26:36.334 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-03T07:26:36.396 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-03T07:26:36.396 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-03T07:26:36.463 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-03T07:26:36.463 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-03T07:26:36.603 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-03T07:26:36.612 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-03T07:26:36.612 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-03T07:26:36.638 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-03T07:26:36.647 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:26:36.702 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2024-06-03T07:26:36.736 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-03T07:26:36.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T07:26:36.792 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-03T07:26:36.801 INFO:teuthology.task.internal:Transferring archived files... 2024-06-03T07:26:36.802 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738514/remote/smithi039 2024-06-03T07:26:36.803 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-03T07:26:37.873 INFO:teuthology.task.internal:Removing archive directory... 2024-06-03T07:26:37.873 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-03T07:26:37.920 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-03T07:26:37.931 INFO:teuthology.task.internal:Not uploading archives. 2024-06-03T07:26:37.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-03T07:26:37.939 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-03T07:26:37.939 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-03T07:26:37.976 INFO:teuthology.orchestra.run.smithi039.stdout: 265304 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 3 07:26 /home/ubuntu/cephtest 2024-06-03T07:26:37.976 INFO:teuthology.orchestra.run.smithi039.stdout: 403334 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 3 05:58 /home/ubuntu/cephtest/mnt.0 2024-06-03T07:26:37.977 INFO:teuthology.orchestra.run.smithi039.stdout: 403335 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 3 06:09 /home/ubuntu/cephtest/mnt.0/client.0 2024-06-03T07:26:37.977 INFO:teuthology.orchestra.run.smithi039.stdout: 403379 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 3 06:09 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-03T07:26:37.977 INFO:teuthology.orchestra.run.smithi039.stdout: 415619 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 3 07:26 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-06-03T07:26:37.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T07:26:37.978 INFO:teuthology.orchestra.run.smithi039.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-06-03T07:26:37.978 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi039 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-06-03T07:26:37.983 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-03T07:26:37.993 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-03T07:26:38.002 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi039 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-06-03T07:26:38.005 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} duration: 5698.407598733902 failure_reason: 'Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi039 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=169a26c25a1c4a16b02980d78a84d1ca status: fail success: false 2024-06-03T07:26:38.006 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-03T07:26:38.096 INFO:teuthology.run:FAIL